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Abstract 

The  purpose  of  this  study  was  to  investigate  and  inopleinent  a  neural  network  rex'  face  verification 
and  classification.  The  research  concentrates  on  developing  a  neural  network  based  feature  extractor 
and/cMT  classifier  which  can  be  used  for  authorized  user  verification  in  a  realistic  woik  envitonmenL 
Performance  criteria  such  as  recognition  accuracy,  system  assumptions,  training  time,  and  execution 
time  were  analyzed  to  determine  the  feasibility  of  a  neural  netwoik  approach.  Specifically,  data  was 
collected  using  a  camcorder  with  automatic  intensity  compensation.  Additionally,  two  segmentation 
schemes  were  used  for  data  collection:  manual  segmentation  and  motion-based,  automatic  segmenta¬ 
tion.  The  data  consisted  of  over  2000,  32x32  pixel,  8  bit  gray  scale  images  of  52  subjects  where  each 
subject  had  from  two  to  ten  days  worth  of  data  collected.  The  data  base  was  then  used  to  create  a 
number  of  training  and  test  sets  that  varied  by  class  size,  training  set  con^sition,  number  of  images  per 
class,  and  so  on.  The  training  and  test  sets  were  then  used  to  train  and  test  the  classification  accuracy  of 
the  following  networks:  a  back  propagation  network  using  the  raw  data  as  inputs;  a  back  prt^gation 
network  using  Karhunen-Lobve  Transform  coefficients,  confuted  from  the  raw  data,  as  inputs;  and  a 
back  propagation  network  using  features  extracted  by  an  identity  network  as  inputs. 

The  results  of  the  various  netwoik  tests  indicate  that  identity  network  extracted  features  perfexmed 
well  for  single  day  captured  data  and  poorly  on  multiple  day  data.  Classification  using  the  various 
networks,  in  general,  performed  well  on  constrained,  single  day  captured,  data  bases  but  performed 
poorly  when  using  data  gathered  over  multiple  days  .  For  the  multiph  day  problem,  a  verification 
network  using  a  single  hidden  layer  with  backpropagation  performed  very  well  and  was  found  to  be 
most  suitability  for  use  in  a  face  verification  system. 
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FACE  RECOGNITION  WITH  NEURAL  NETWORKS 


/.  Problem  Desaription 


LI  Introdactioa 

Autonomous  face  recognition  is  the  (vocess  of  locating  and  identifying  faces  in  a  scene  using 
pattern  recognition  techniques.  While  humans  recognize  faces  many  times  a  day  with  a(^)arent  ease, 
automating  this  (nxx»ss  has  challenged  researchers  for  die  past  two  decades.  What  does  an  automated 
face  recognition  system  offer  us  to  wanant  the  years  of  research  this  problem  has  received? 

A  system  that  automatically  recognizes  faces  would  be  useful  for  several  reasons.  From  a  security 
perspective,  an  automatic  face  recognition  system  could  enhance  current  access  control  systems  by 
authenticating  a  users  identity  (32).  Exanqiles  of  such  access  control  systems  are  secure  computer 
systems,  bank  automatic  teller  machines,  and  automatic  card  readers.  In  fact,  any  (nganization  or 
system  that  permits  access  based  on  a  person’s  identity  would  find  a  face  recognition  system  useful 
(32).  Other  security  qiplications  fw  a  face  recognition  system  would  be  criminal  identification  and 
scanning  airports  for  terrorists.  Finally,  this  system  could  be  adrqpted  for  use  in  a  speedi  recognition 
system  ot  a  visual  communications  system  (32). 

The  remainder  of  this  problem  description  begins  with  a  background  review  of  face  recognition 
as  it  relates  to  pattern  recognition  and  this  thesis.  i  nis  is  followed  by  die  problem  statement,  research 
objectives,  assumptions,  scope  and  limitations,  and  standards.  The  approach  to  die  problem  is  then 
discussed  and  the  cluqiter  concludes  with  an  overview  of  the  remaining  chapters. 
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1.2  Background 


This  section  briefly  discusses  the  pattern  recognition  process  and  then  highlights  some  of  the 
current  research  in  the  area  of  autonomous  face  recognition.  The  research  highlighted  below  will  be 
discussed  in  detail  in  chapter  two  but  is  included  here  for  completeness. 

Traditionally,  pattern  recognition  is  broken  down  into  three  areas:  segmentation,  feature  extrac¬ 
tion,  and  classification  (38).  Segmentation  is  the  first  step;  it  is  finding  regions  of  possible  sigiuils.  The 
second  step  is  feature  extraction  and  in  this  step  we  search  for  the  nwst  important  or  significant  features 
of  the  regions  passed  by  the  segmentor  which  can  be  used  in  the  final  step,  classification.  CHassification 
compares  the  extracted  features  to  those  of  previously  identified  objects  and  identifies  the  object  as  one 
of  the  previously  identified  classes.  For  this  research  the  regions  we  desire  to  segment  and  identify  are 
faces  but  the  approach  is  the  same  whether  we  are  using  faces,  trucks,  tanks,  jeeps,  ot  words  in  speech. 

1.2.1  Ttirk  and  Pentland  at  the  MIT  Media  Lab.  Matthew  Tuiic  and  Alex  Pentland,  from  the 
Massachusetts  Institute  of  Technology  Media  Lab,  have  inqrlemented  an  autonorrxHis  face  recognition 
system  that  also  makes  use  of  the  Karhunen-Lodve  Transform  (40).  While  their  system  performs  well, 
it  has  limited  application  because  of  the  enormous  amount  of  cotiqrutational  power  necessary  to  operate 
the  system — at  present,  the  system  requires  three  dedicated,  high  speed  processors. 

1.2.2  Face  Recognition  at  AFIT.  Face  recognition  at  AFTT  began  in  1985  with  early  systems 
that  were  slow  and  required  a  significant  amount  of  human  intervention  (31,  35).  Through  die  years, 
various  improvements  were  added  and  the  system  has  evolved  into  what  it  is  today  (18, 33  27)  The 
current  system  is  similar  to  the  MIT  system  being  developed  by  Turk  and  Pentland  (36).  However,  die 
AFTT  system  has  one  major  improvement  over  the  MIT  systera  That  is,  the  current  AFTT  preprocessor 
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incorporates  a  centering  routine  and  a  gaussian  window  routine.  These  routines  center  die  image 
and  draw  a  gaussian  window  around  the  image,  thereby  de-enqihasizing  die  background.  With  diis 
improvement,  the  current  AFTT  system  is  95  percent  successful  at  identifying  55  face  images  (36).  The 
current  AFTT  system  is  used  as  a  benchmark  for  this  research  effort 

1.2.3  Cottrell  at  UCSD.  Garrison  Cottrell  of  the  University  of  California  at  San  Diego  has 
been  developing  a  neural  network  model  approach  to  face  recognition  (11).  His  model  is  trained  in  a 
two  step  approach  that  uses  an  identity  network  for  feature  extraction  and  a  single  layer  baclqiropagation 
network  for  classification.  He  has  shown  this  network  to  be  successful  on  limited  databases. 

1.2.4  Los  Alamos  National  Laboratory.  Another  neural  networic  approach  to  face  recogni¬ 
tion  is  being  pursued  at  the  Los  Alamos  National  Laboratory  (25).  The  lab  uses  a  standard  one-hidden 
layer  network  trained  by  backptopagation  using  raw  image  data  as  inputs.  (Currently,  the  networic  is 
limited  to  two  class  problems  on  a  constrained  data  base  but  the  preliminary  results  are  very  respectable. 

1.3  Problem  Statement 

This  research  effort  will  focus  on  improving  the  classification  performance  and  speed  of  die 
autonomous  face  recognition  system  by  implementing  the  feature  extraction  and  classification  fdiases 
cf  the  recognizer  in  a  neural  network.  The  segmentation  phase  of  the  recognizer  is  being  researched  in 
a  collateral  thesis  by  Obtain  Kevin  Gay  (13). 
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1.4  Research  Objectives 


The  objective  of  this  research  is  to  improve  the  algorithm  for  feature  extraction  and  classification 
of  faces  in  an  autonomous  face  recognition  system.  Specifically,  four  netw^k  configurations  will  be 
developed  and  con:q)ared; 

•  An  identity  network  for  data  compression  and  feature  extraction, 

•  A  back  propagation  network  using  the  raw  data  as  inputs, 

•  A  back  propagation  network  using  Karhunen-Lodve  Transform  coefficients,  con^Mited  from  the 
taw  data,  as  inputs, 

•  And  3-layer  and  4-Iayer  feature  extraction  and  classification  networks  using  back  propagation 
and  raw  data  inputs. 

1.5  Assumptions 

•  The  distance  from  the  subject  to  the  camcorder  is  constant,  with  die  exception  of  normal  head 
movement 

•  A  friendly  user  is  assumed,  ie.  the  head  orientation  is  fece  front 

•  The  netwmk  is  expected  to  run  on  a  Sun  wtakstation. 

•  If  using  automatic  segmentation .. . 

-  the  face  has  been  properly  segmented  from  the  background. 

-  preprocessing  to  center  and  gaussian  window  the  face  has  been  accomplished. 

•  If  using  manual  segmentation .. . 
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-  the  face  has  been  manually  centered  in  the  image. 


-  preprocessing  to  gaussian  window  the  face  has  been  accon^Iished. 

1.6  Scope  and  Limitations 

The  scope  of  this  thesis  is  to  investigate  the  limitations  of  several  new  algorithms  used  fcM- 
autonomous  face  recognition.  All  conclusions  are  based  on  test  data. 

1.7  Standards 

The  performance  criteria  for  the  algorithms  are  classification  accuracy,  user  interaction,  and 
modularity.  However,  accuracy  is  the  most  important  of  these  criteria. 

1.8  Approach/Methodology 

A  software  environment  will  be  developed  and  executed  on  a  Sun  SPARCstation2  that  combines 
existing  software  with  new  software  written  in  ANSI  C.  AH  algotiduns  will  be  devel<q>ed  with  mod¬ 
ularity  as  a  key  consideration.  Data  for  training  and  testing  will  be  gathered  under  as  many  varying 
conditions  as  possible  to  test  the  robustness  of  the  algorithms. 

1.9  Overview 

Qiapter  Two  presents  a  review  of  current  literature  related  to  face  recognition  systems  with 
primary  enqrfiasis  on  feature  extraction  and  classification.  Chapter  Three  provides  a  detailed  description 
of  the  methodology  used  in  tiiis  thesis,  and  Quarter  Four  provides  test  criteria  and  results.  Chapter 
Hve  presents  conclusions  based  on  die  test  results  and  makes  recommendations  for  future  study. 
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//.  Literature  Review 


2.1  Introduction 

This  review  examines  some  of  the  current  literature  in  the  area  of  autononoous  face  recognition. 
Face  recognition  research  over  the  past  several  years  falls  into  two  categories:  recognition  using 
features  and  recognition  using  the  whole  face  or  holistic  approach. 

Face  recognition  using  features  was  first  attempted  by  L.  D.  Harmon  in  the  early  1970s.  He 
extracted  features  from  profiles  to  identify  the  faces.  His  features  were  defined  as  the  distance  fnmi  die 
tip  of  the  nose  to  the  mouth,  the  distance  from  the  nose  to  die  chin,  the  distance  from  die  eyes  to  the  nose 
and  other  similar  measurements  (16).  In  addition  to  Harmon’s  method,  odier  types  of  face  recognition 
using  features  involves  segmenting  a  face  and  then  extracting  features  from  die  segments.  Whatever 
the  method,  face  recognition  using  features  continues  today  with  researchers  all  over  the  world  (42, 2). 
The  second  category  of  face  recognition  is  the  holistic  tqqxroach.  Research  in  face  recognition  has 
moved  towards  a  holistic  point  of  view  with  researchers  at  the  Massachusetts  Institute  of  Technology 
(MIT)  (40, 39),  the  University  of  California  San  Diego  (UCSD)  (11,5)  and  AFTT  (36,  IS).  The  holistic 
approach  still  involves  extracting  features,  but  the  features,  which  are  extracted  using  some  type  of 
principal  con^nent  analysis,  are  now  taken  from  the  entire  face  image,  not  just  segments  ot  profiles. 
This  research  is  based  on  the  holistic  i^iproach  and  it  is  what  will  be  discussed  in  the  following  sections. 

The  remainder  of  this  review  begins  with  some  Ixief  biological  notes  on  human  face  rectignititni. 
This  is  followed  by  a  discussion  of  feature  extraction  and  data  compression  using  principal  component 
analysis  and  neural  netwoiks.  The  review  continues  widi  discussions  on  several  networks  used 
for  face  rect^nition  whidi  include  the  Cottrell  netwcak,  a  standard  multiple  layer  perceptron  using 
backpropagation,  and  the  Tarr/Ruck  network  for  Kaihunen>Loive  transformations  and  classification. 
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Finally,  several  anecdotes  are  discussed  that,  while  not  directly  related  to  diis  research,  are  still  part  of 
the  face  recognition  literature  and  should  be  mentioned. 

2.2  The  Bkdogy  of  Face  Recognitioa 

How  does  the  brain  acconq>lish  face  recognition?  This  is  sdll  a  mystery  but  we  are  learning 
more  each  day.  Experts  believe  that  the  biological  process  of  face  recognititm  occurs  on  the  underside 
of  both  hemispheres  of  the  brain  in  the  tenqxMal  and  occipital  lobes  (14).  In  odm^  words,  diis  process 
is  localized  (found  in  a  specific  area)  in  our  brains.  This  theory  is  supptHled  by  die  work  of  Rolls, 
Baylis,  Hasselmo  and  Nalwa  in  their  study  of  the  response  of  specific  neurons  to  faces  (28).  They 
tested  a  group  of  face  responsive  neurons  in  macaque  monkeys  and  recorded  die  neuronal  responses 
(action  potential  spikes  per  second)  to  a  set  of  face  stimuli  and  non-face  stinuili.  The  results  of  their 
study  indicate  that  there  are  neurons  that  react  primarily  to  fu»s.(28). 

Localization  is  also  supported  by  the  work  of  J.  C.  Meadows  and  A.  R.  Damasio  in  dieir  studies 
of  individuals  who  have  lost  the  ability  to  recognize  faces,  a  condition  known  as  prosopagnosia.  Both 
researchers  agree  that  patients  with  prosopagnosia,  when  they  have  come  to  autopsy,  always  have 
bilateral  lesions  in  the  occipito-temporal  regions  of  their  brains  (8,  20).  Conversely,  patients  widi 
bilateral  lesions  in  other  portions  of  the  Inain  (ie.,  occipito-parietal  region)  do  not  have  prosopagnosia. 
This  also  supptwts,  albeit  in  a  limited  manner,  the  notion  that  the  (vocess  is  localized.  Accepting 
localization  as  fact,  what  information  does  this  area  of  the  brain  require  to  recognize  faces? 

One  theory,  based  on  psychological  experimentation,  suggests  diat  we  store  the  information  for 
faces  in  a  global  to  local  scheme  (41).  In  dus  experiment  subjects  were  first  asked  to  class^  images 
as  faces  or  non-faces  and  die  time  to  classify  faceness  was  recorded.  Then  each  subject  was  asked 
to  ident^  familiar  faces  which  were  either  distinctiveAinusual  or  tyincal  faces.  It  was  found  diat 
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faceness  could  be  assessed  very  quickly  while  recognition  took  longer.  Also,  a  distinctive  face  could 
be  identified  more  quickly  than  a  typical  face.  As  such,  it  was  [noposed  that  the  d^erence  information 
(ie. ,  color  of  eyes,  shtqw  of  nose,  hair  style,  etc. )  is  used  in  the  identification  process  and  the  padi. 
Information  that  is  the  same  (ie. ,  the  general  shape  of  a  face,  two  eyes,  two  ears,  etc. )  is  used  at  a  higher 
level  in  the  recognition  process  and  is  not  stored  fcs’  each  particular  face.  This  could  be  ctmsidered  a 
global  to  local  type  process  which  fits  our  human  experience  with  face  recognition  because  we  often 
say  that  a  person  looks  like  so  and  so  except  f(»'  their  eyes  or  nose,  etc.  This  global  to  local  idea  is 
also  supported  in  the  prosopagnosia  studies.  Individuals  widi  prosq>agnosia  can  still  identify  a  face  as 
a  face,  but  they  can  not  identify  who  the  face  belongs  to.  In  fact,  many  prosopagnosia  patients  also 
have  difficulties  with  recognition  of  other  objects  such  as  a  particular  make  of  car  or  truck.  As  with  the 
faces,  they  can  identify  a  car  as  a  car  but  they  can  not  tell  what  specific  type  of  car  it  is  (8).  This  implies 
that  we  ]m>cess  general  information  first  and  then  get  down  to  the  specifics  for  the  actual  identification 
process,  but  what  are  the  specifics? 

Experts  agree  that  something  is  stored  or  encoded  in  the  face  recognition  neurons;  however,  they 
disagree  as  to  what  that  something  is.  One  theory  is  that  die  iiif(»mation  for  a  particular  face  is  stored 
in  a  grandmother  cell  (28).  In  odier  words,  all  the  infnmation  for  a  particular  person  is  encoded  in  a 
single  cell  (neuron)  and  when  we  find  that  cell  we  know  who  the  person  is.  A  second  dieoiy,  supported 
by  the  wmit  of  Rolls  and  his  colleagues  is  that  the  infcxmation  for  recognizing  a  face  is  stored  in  a  coded 
ensemble  of  neurons  and  diese  codes  are  reconqxited  whenevn-  a  new  face  is  added  to  the  system  (28). 
This  the«y  was  tested  by  recording  the  activity  of  face  responsive  neurons  to  a  set  of  known  faces  and 
then  adding  a  novel  face  to  the  set  and  recording  the  responses  to  this  new  set  The  tests  indicated  diat 
adding  a  new  face  alters  the  steady  state  response  of  some  of  die  neurons  in  question  for  a  short  time 
and  dien  diey  reach  a  new,  statistically  different,  steady  state  (28).  This  implies  diat  information  for 
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face  recognition  is  stored  in  our  face  neurons  and  we  know  who  an  individual  is  based  ra  die  firing 
pattern  of  this  group  of  neurons.  This  also  suggests  that  the  entire  code  is  changed,  albeit  very  quickly, 
each  time  a  new  face  is  added  to  the  systeia 

In  any  case,  the  biological  theories  surrounding  face  recognition  are  as  varied  as  die  nundier  of 
researchers.  The  accepted  theory  today  is  that  there  are  specific  face  recognition  neurons  in  our  brains 
and  the  global  (faceness)  to  local  (identification)  idea  is  gaining  momentum.  Rrom  a  pattern  recognition 
point  of  view,  this  global  to  local  idea  could  be  considered  data  coirqiression,  which  is  important  to  the 
design  of  a  face  recognition  system. 

2.3  Feature  Extraction  and  Data  Compression 

Feature  extraction  and  data  compression  are  important  problems  in  pattern  recognition  and  image 
analysis.  Many  times  the  goal  is  to  find  a  set  of  features  that  represent  the  data  as  closely  as  possible  and 
compress  the  data  at  the  same  time.  There  are  many  approaches  to  this  problem  and  me  of  die  more 
well  known  is  Principal  Conqionent  Analysis  (PCA).  Likewise,  diis  problem  has  been  addressed  in 
linear  and  nonlinear  PCA  neural  networks  and  in  autoassodation  networks  as  an  unsupervised  learning 
task  (24).  Each  of  these  will  be  discussed  in  the  following  sections. 

2.3.1  Principal  Component  Analysis  (PCA).  In  general  terms,  PCA  is  a  statistical  method 
used  for  extracting  features  from  a  set  of  data  with  high  dimensionality.  It  is  a  solution  to  die  curse  of 
dimensionality  problem  found  in  pattern  recognition  (9).  PCA  is  a  linear,  ordiogonal  transformation  or 
projection  of  die  data  onto  a  new  coonfinate  system  whoe  die  axis  are  uncorrelated  and  die  maximum 
variance  of  the  original  data  is  found  in  only  a  small  number  of  coordinates  (12).  Dimensionality 
reduction  is  adiieved  in  diis  space  by  taking  the  coordinates  which  have  the  maximum  variance  and 
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leaving  out  the  coordinates  with  minimum  variance.  In  mathematical  terms,  the  basis  vectors  of  the 
new  cotmlinate  system  are  the  eigenvectors  of  the  covariance  matrix  of  die  data  and  the  variances 
are  the  corresponding  eigenvalues  (40).  So  in  terms  of  PCA,  the  best  projection,  in  terms  of  mean 
squared  error  of  reconstruction,  from  an  Af  to  an  jyf  dimensional  space,  where  M  >>  N,  is  then  die 
N  dimensional  space  which  represents  the  N  eigenvecttws  with  the  largest  eigenvalues. 

Researchers  at  the  MIT  Media  Lab  and  AFTT  use  principal  congionent  analysis  in  dieir  holistic 
approach  to  face  recognition  because  it  is  believed  that  specific  features,  such  as  eyes  or  nose,  may  not 
be  as  inqioitant  as  the  overall  pattern  of  the  face  when  it  comes  to  recognition.  A  detailed  review  of 
PCA  using  the  Kariiunen-Lotfve  transfmm  can  be  found  in  a  masters  diesis  by  Pedro  Suarez  (36).  This 
tqiproach  is  suppcxted  by  the  physiology  and  psychology  of  die  face  recognition  process  (40, 41 ).  While 
the  above  process  is  mathematically  sound,  calculation  of  eigenvectors  and  projection  coefficients  is 
conqmtationally  expensive.  As  such,  the  MIT  face  recognition  system  is  nm  using  three  cotqwters: 
a  Datacube  Image  Processor,  a  Sun  3/160,  and  a  Sun  Spaicstation.  This  system  can  perform  die 
recognition  task  at  a  rate  of  two  or  diree  times  a  second  (40)  but  the  system  size  makes  it  impractical 
for  current  tqiplications.  Like  die  MIT  system,  die  AFTT  system  is  hosted  on  multiple  ctmqiuters. 
The  preprocessing  software  is  hosted  on  a  NEXT  computer  and  a  Silicon  Graphics  computer,  and  die 
remainder  of  the  software  runs  on  a  second  Silicon  Gr^ihics  computer.  However,  even  widi  multiple 
computers,  the  fnocess  still  takes  several  minutes  to  run.  The  question  becomes  bow  do  we  improve 
die  speed  of  the  system  without  giving  up  any  of  the  accuracy?  The  answer  may  be  a  neural  network 
iqiptoach  to  PCA. 

2.3.2  PCA  using  a  Linear  Ncnrai  Nctimrli.  When  considering  a  neural  network  qiproadi 
to  PCA  a  starting  point  in  mudi  of  the  literature  is  die  Qja  algoridim  (22).  His  network  model  shown  in 
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Figure  2. 1.  consists  of  a  single  linear  neuron  unit  diat  uses  a  Hebbian  type  of  learning  rule.  He  showed 
that  if  the  input  vectors  are  a  stochastic  process  this  network  tends  to  extract  the  hugest  principal 
component  from  the  input  vectors.  This  corresponds  to  the  eigenvector  in  the  covariance  matrix  diat 
has  the  largest  eigenvalue. 


Figure  2.1.  Oja  linear  PCA  network  which  converges  to  die  largest  principal  con^nent  of  a  stationary 
input  sequence  (12). 

The  ouqiut  of  Oja’s  net,  y,  and  the  learning  rule  for  updating  the  weights,  9,  ,  are 

y  =  ^giXi  (2.1) 

»=i 

Aqi  =  p{xiy  -  qiy^)  (2.2) 

where  x,y  is  the  Hebbian  term  diat  strengdiens  the  connectimis  when  the  iqxit  and  the  output  are 
correlated.  Thesecondterm,— 9,1/’,  is  used  to  prevent  instability  and  makes  ^^qiproadi  1.  Training 
the  net  in  this  manner  maximizes  the  variance  of  the  output  given  die  constraint  that  9J  =  1.  The 


2-6 


disadvantage  of  the  netwcnic  is  that  it  will  only  find  the  first  principal  component  of  die  data  set  (12) 
and  in  many  cases  more  principal  conqxinents  ate  requited  in  (nder  to  be  useful  fcv  a  given  problem. 

The  research  continued  and  sevetal  algoriduns  have  been  developed  diat  find  multiple  principal 
components  of  a  set  of  data;  one  of  the  more  recent  algoriduns  is  the  Adiqitive  Rnncipal-conyxinent 
Extractor  (APEX)  (17).  APEX  was  pnqxised  by  Kung  and  Diamantaras  in  1990  and  is  stated  to 
be  recursive  and  adaptive  in  that,  given  the  first  m  —  1  principal  components,  it  will  find  die 
component  Additionally,  diey  show  that  die  component  is  die  largest  component  which  is 
orthogonal  to  the  previous  m  —  1  conqxinents.  Their  networic,  shown  in  Figure  2.2,  combines  die 
properties  of  Oja’s  linear  PCA  net  with  a  deconrelation  scheme  proposed  by  Fbldiak  in  (12)  that  causes 
the  network  to  woric  as  a  whitening  filter. 


Hgure  2.2.  APEX  netwwk:  solid  lines  denote  weights  p,-,  Wj  and  are  trained  at  the  stage.  (Note 
that  Wj  asyn^totically  approach  zero  as  the  network  converges)  (17). 


The  ouQMits  of  the  network  are 


y  =  Px 


y„=px  +  wy 


(2.3) 

(2.4) 
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where  z  is  the  input  vector,  y  is  the  ou^ut  vector  representing  die  first  m  —  1  ou^t  ccMi^nents,  P 
is  the  matrix  of  pij  weights  of  the  first  m  —  1  conqxinents,  and  p  is  the  row  vector  of  the  Pmj  weights 
of  the  mth  output 

The  equations  for  the  nith  component  then  become 

Ap  =  -  ylfi)  (2.5) 

Aw  =  -7(ymy^  +  ym*")  (2-6) 


where  p  and  7  are  positive  learning  rates. 

Hnally,  if  the  above  equations  are  expanded  for  each  individual  weight  die  resulting  equations 
are 

Apj,j  =  /3(ym® j  -  =  1 . . .  n  (2.7) 

Awj  =  -'riymVj  +  ymWj)J  =  1 ...  m  -  1.  (2.8) 

A  review  of  these  equations  shows  that  equation  2.5  is  simply  Oja’s  hebbian  update  rule  whidi 
was  shown  in  (17)  to  force  the  ouqiuts  to  the  dominant  principal  conqionents,  and  equation  2.6  is  an 
anti-hebbian  rule  which  was  shown  to  cause  the  mth  ouqmt  to  be  ordiogonal  to  (ot  uncorrelated  firran) 
the  previous  m  —  1  con^nents.  Results  of  tests  of  APEX  indicate  that  the  i»incipal  conqxments  it 
extracts  are  almost  peifecdy  normalized  and  ordiogonal  to  one  another  and  are  very  close  to  the  actual 
conqionents  found  using  statistical  PCA  (17). 

In  addition  to  APEX,  other  algorithms  for  use  in  linear  neural  networks  have  been  proposed 
for  PCA.  Sanger  (34)  proposed  an  algoridim  diat  uses  non-local  information  which  complicates  the 
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analysis  and  Foldiak  (12),  mentioned  earlier  produces  a  set  of  vectors  diat  span  the  same  space  as  the 
principal  comqwnents  but  are  not  the  exact  principal  con^nents  (1).  Additionally,  an  algoritiun  based 
on  Successive  Application  of  Modified  Hebbian  (SAMH)  learning  was  proposed  recently  (1)  which  is 
shown  to  extract  the  principal  components  in  an  adaptive  manner,  similar  to  APEX,  but  is  claimed  to 
converge  much  more  quickly. 

PC  A  in  linear  networks  is  also  supported  by  the  work  of  Baldi  and  Hotnik  who  showed  that  the 
error  surface  for  this  linear  type  netwoik  has  a  unique  minimum  that  corresponds  to  the  projection  onto 
the  subspace  generated  by  the  (xincipal  conponents  of  the  input  data  set  (3).  However,  they  also  state 
that  the  optimum  solution  using  principal  components  could  also  be  obtained  using  other  well-known 
algorithms  for  conq>uting  eigenvalues  and  eigenvectors,  and  by  numerical  analysis  standards,  diese 
algorithms  are  superior  to  using  a  linear  neural  netwtMk  to  extract  principal  conqwnents  (3).  Likewise, 
given  data  sets  containing  outliers,  the  linear  networks  have  been  shown  to  degenerate  (19).  Widj 
this  in  mind,  why  not  disregard  the  notion  of  back  propmgation  neural  netwt^cs  fcH’  feature  extraction 
altogether?  The  answer  in  addition  to  the  simplicity  of  error  back  propagation,  it  can  be  applied  to 
nonlinear  networks,  discussed  in  the  next  section,  which  have  been  shown  to  handle  the  outlier  problem 
(19,  6).  Additionally,  it  is  very  successful  in  a  variety  of  other  problems  where  tiiere  is  no  a  priori 
knowledge  of  the  structure  of  the  mathematical  properties  of  the  ideal  solution  (3). 

2.3.3  Feature  Extraction  and  Data  Compression  in  Nonlinear  Networks  In  applications 
like  pattern  recognition  and  speech  recoguition  an  important  problem  is  to  find  die  relevant  features 
in  order  to  cotiqness  die  data  and  still  allow  for  ctKiect  classification  or  representation  of  die  data 
(23).  A  reasonable  requirement  in  data  comfRession  is  that  the  original  pattern  can  be  restored  from 
the  cotiqxessed  data  widi  some  acceptable  level  of  error.  This  iiigilies  that  feature  extraction  is  data 
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driven  and  independent  of  any  pattern  and  in  terms  of  neural  netwtnics,  this  is  an  unsiqiervised  or 
selfsupervised  learning  task  (23).  One  approach  to  the  feature  extraction  problem  has  been  the  use 
of  nonlinear  autoassociation  neural  networks  which  have  well-known  approximation  prc^rties,  can 
be  expected  to  improve  performance  (23)  and  can  solve  the  encoding  problems  where  strict  principal 
component  analysis  is  degenerate  (6).  These  networks  are  generally  feedforward  3-layer,  Hgure  2.4, 
or  5-layer,  Figure  2.3. 

The  S-layer  network  of  Rgure  2.3  is  believed  to  be  more  robust  because  it  can  die(»etically 
conqrute  any  continuous  mapping  from  inputs  to  the  second  hidden  layer  and  another  mapping  from 
the  second  hidden  layer  to  the  output.  The  first  and  drird  );>yers  are  nonlinear  (usually  sigmoidal) 
and  layers  one,  three,  and  five  are  linear.  The  ouqrut  of  layer  three  (p  units),  can  now  be  used  as  the 
extracted  features  which  are  inputs  to  a  classifier.  The  mapping  from  layer  one  to  two  can  be  considered 
a  nonlinear  PCA,  and  the  mapping  from  two  to  three  can  be  considered  a  linear  PCA.  In  this  case  the 
dimensionality  was  reduced  from  n  to  p.  However,  a  significant  problem  with  this  network  is  that  the 
number  of  units  in  the  nonlinear  layers  must  be  large,  N  »  n,  in  (»der  to  get  good  rq^rroximation 
capabilities  for  reconstruction  (23).  This  constraint  can  be  a  problem  for  any  network  when  training 
time  is  considered,  especially  if  the  number  of  inputs  is  already  large^ . 

An  alternative  to  the  S-layer  network  is  the  nonlinear  3-layer  network  of  Figure  2.4  which  is 
also  one  solution  to  the  training  time  problem  of  a  S-layer  network.  The  3-layer  nonlinear  netwtwk  has 
hidden  units  that  develop  weight  vectors  that  are  believed  to  span  die  principal  subspace  of  die  input 
vectors,  ie.  they  develop  a  distributed  representation  of  the  principal  con^nents  (7).  This  result  was 
found  en^irically  by  presenting  the  eigenvectors  of  die  input  images  as  inputs  to  die  network  (6).  In 

'For  example,  assume  the  input  image  is  32x32  (1024  pixelsX  the  second  layer  and  fourth  layers  of  net  are  also  1024 
nodes  (not  larger  as  is  suggested),  and  the  third  layer  is  only  40  nodes,  this  network  is  estimated,  conservatively,  to  require 
80  days  to  train  (based  on  current  tun  times  for  similar  networks  on  Sun  workstations,  see  chapter  4). 
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Figure  2.3.  S-Iayer  network  with  linear  and  nonlit>ear  layers.  The  number  of  units  in  a  layer  is  given 
above  the  box,  with  N  »  n  >  p,  and  the  output  is  given  below  the  box.  (23). 


Figure  2.4.  3-layer  network  with  nonlinear  hidden  and  ou^t  layers.  The  number  of  units  in  a  layer 
is  given  above  the  box  and  the  ouq>ut  is  given  below  the  box. 


other  words,  if  the  network  is  treated  as  a  matrix  M,  where  M  represents  the  learned  weights  of  the 
input  layer,  then  find  E'  such  that 

E'  =  ME  (2.9) 

where  £7  is  the  matrix  formed  by  the  k  =  AT,  Karhunen-Loive  generated  eigenvectors  of  file  input 
image  in  descending  eigenvalue  order  as  the  columns.  If  the  network  spans  the  principal  subspace  of 
dimension  k,  then  E  and  E'  ate  related  by 


I  =  EE^ 


(2.10) 
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where  l'  is  the  identity  matrix.  The  results  of  this  technique  show  a  noisy  first  component  with  the  rest 
of  the  components  being  slightly  shortened  (ie.  the  diagonal  entries  of  /'  are  slightly  less  than  1)  (6). 
It  is  believed  that  the  hidden  units  develop  a  distributed  representation  of  the  principal  conqmnents  of 
the  input  data  because  the  variances  of  the  output  activations  of  the  hidden  units  are  distributed  more 
evenly  than  the  variances  of  the  components  in  the  strict  Karfaunen-Lo&ve  transform  (6). 

It  has  been  suggested  that  reducing  the  dimensionality  with  a  3-layer  network  is  no  better 
than  using  the  standard  Kaihunen-Lo&ve  transform  (21),  however,  this  net  has  been  used  for  image 
compression  with  good  results  (23,  5,  6,  7)  and  it  is  believed  that  the  advantages  of  the  nonlinearity 
will  come  out  when  the  problem  is  nonlinear  (6). 

2.3.4  Karhunen-Lo^ve  Network  for  Feature  Reduction  A  network  was  developed  by  (jre- 
gory  Tarr  and  others  at  the  Air  Force  Institute  of  Technology  that  performs  feature  reduction  using  a 
variation  on  the  ICariiunen*Lo&ve  transform  (37).  The  network,  shown  in  Figure  2.5,  is  described  in 
detail  in  (37)  but  the  idea  is  as  follows.  Every  random  vector,  X,  in  a  data  set  can  be  represented  by  a 
linear  transformation  of  X  with  a  matrix.  A,  such  that 

X  =  AY  (2.11) 

where  A  is  contused  of  the  normalized  eigenvectors  of  the  data,  X,  covariance  matrix  in  descending 
eigenvalue  order  and  A  does  a  rotation  in  the  vector  space  of  X  to  the  new  vector  Y . 

If  the  columns  of  A  are  orthonormal  and  form  n  linearly  independent  basis  vectors,  then  die 
following  conditions 

A^ A  =  I  and  A~‘  =  A^  (2.12) 
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Hgure  2.5.  Network  Architecture  for  Karfaunen-Lodve  Feature  Extractor.  (37) 


allow  the  vector  V  to  be  written  as 

y  =  A^X  (2.13) 

where  the  dinnensionality  of  Y  is  equal  to  the  dimensionality  of  X,  and  y  is  die  vectra’  of  Kaihunen- 
Loive  coefficients  found  from  the  rotation  of  X  to  Y.  If  the  goal  is  to  reduce  die  dimensionality  of  Y, 
and  if  for  all  X  presented  to  the  network,  some  of  the  nodes  of  Y  were  very  small,  zero,  or  constant, 
then  they  would  not  be  necessary  to  reproduce  an  esdmate  of  X : 

m  fi 

Mm)  =  ^yiAi+  x;  M.  (2.14) 

i=l 

where  6,  coefficients  are  independent  of  X  and  y,  coefficients  are  dependent  on  X  (37).  Now,  if  die 
objective  is  to  classify  and  not  reproduce  X,  then  since  the  value  b,-,  t  =  m  +  1 . . .  n,  is  constant  or 
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nearly  zero,  it  can  be  ignored.  The  number  of  eigenvectors  chosen  from  A  is  dien  m  and  the  network 
has  effectively  reduced  the  dimension  of  the  data  set  Likewise,  since  we  are  using  PCA  to  reduce  die 
dimension  we  have  extracted  the  primary  features  which  can  now  be  used  for  classification  of  die  data. 
The  classification  is  performed  by  the  top  portion  to  the  network,  see  Hgure  2.S.  It  should  also  be 
noted  that  in  order  to  make  the  network  training  easier  the  data  must  be  statistically  nmnialized  before 
presentation  to  the  feature  reduction  portion  of  the  netwmk  and  then  statistically  normalized  once  more 
befme  presentation  to  the  classification  portion  of  the  networic  (37). 

2.4  Neural  Networks  for  Face  Recognition 

2.4. 1  Cottrell  Neural  Network.  A  neural  network  for  face  recognition  is  being  developed  by 
Garrison  Cottrell  at  the  University  of  California  at  San  Diego  (11, 5).  His  network,  shown  in  Figure 
2.6,  is  trained  using  back  propagation  and  is  composed  of  his  image  compression  (auto-association) 
network  and  a  single  layer  perception.  The  parameters  for  the  networks  in  Figure  2.6  are  as  follows: 

•  Auto-associator  Network 

-  Input  nodes:  4096 

-  Hidden  nodes:  40,  sigmoid  activation  range  [-1,1] 

-  Ou^ut  nodes:  4096,  sigmoid  activation  range  [-1,1] 

-  Momentum:  0 

-  Hidden  layer  learning:  0.0001 

-  Ou^nit  layer  learning:  0.1 

•  Classification  Netwrxk 
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-  Input  nodes:  4096 


-  Hidden  nodes:  40,  sigmoid  acdvadon  range  [-I,!] 

-  Ouq)ut  nodes:  20,  sigmoid  activation  range  [-1,1] 

-  Momentum:  0 

-  Hidden  layer  learning:  Fixed  weights 

-  Output  layer  learning:  0.1 

The  basic  operation  of  the  network  is  as  follows.  First,  the  auto-associatra'  networic  is  trained  to 
match  output  images  to  input  images.  The  images  {<x  this  experiment  were  64x64  fHxels,  txightness 
normalized,  and  manually  centered.  There  were  eight  images  for  each  of  20  students,  10  male  and  10 
female,  making  a  total  of  160  images.  The  auto-associator  netwwk  used  ftM*  this  phase  consists  of 4096 
(64x64)  input  nodes,  40  hidden  nodes,  and  40%  output  nodes.  Successful  training  is  defined  as  the  root 
mean  square  pixel  intensity  error  rate  being  less  duui  12  gray  levels  between  input  and  outfNit  which 
corresponds  to  an  average  squared  error  per  unit  of  .0017.  The  enrra*  rate  is  met  in  approximately  SO 
epochs  which  indicates  that  the  netwcuk  is  trainable  in  a  slxxt  amount  of  time.  Dr.  Cottrell  dieotizes  that 
the  hidden  layer  conq)resses  the  data  and  extracts  features  that  reinesent  a  distributed  representation  of 
the  principal  conq>onents  which  are  similar  to  die  eigenfaces  of  the  MIT  face  recognition  system.  He 
supports  this  theory  by  forming  the  covariance  matrix  of  the  hidden  unit  activations  over  all  images  and 
extracting  the  principal  components  using  strict  (nincipal  cotiqionent  analysis.  The  resulting  principal 
components  ate  then  decom(»essed  by  running  diem  through  the  output  layer  of  die  auto-associadon 
network.  Rguie  2.7  shows  images  reconstructed  in  this  manner,  and  diey  are  similar,  at  least  in 
appearance,  to  the  MIT  eigenfaces. 
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Auto-association  Network 


Classificadon  Netwcnk 


4096  Output  nodes  20  Output  nodes 


4096  Input  nodes  4096  Input  nodes 


Figure  2.6.  Left  is  the  auto-associadon  networic  used  to  train  die  input  layer  weights  for  the  netwoik 
on  the  right  The  ouq)ut  layer  weights  on  the  classification  network  are  traiited  using 
baclqHX)p.  (S) 


After  the  auto-associator  network  is  trained,  the  weights  between  the  input  and  hidden  layer 
are  fixed  and  the  ouQ)ut  of  the  hidden  layer  is  connected  to  a  smaller  (20  ouqnit  nodes),  single-layer 
classification  network.  This  smaller  netwmk  is  then  trained  to  classify  (identify)  each  image;  dte 
netwoik  is  also  trained  to  identify,  and  classify  as  ‘unknown’,  nonface  images  at  this  time.  The  test 
results  for  this  netwoik  were  99  percent  recognition  accuracy.  However,  the  data  base  used  was 
constrained  to  20  subjects  and  all  images  were  captured  on  the  same  day  and  time.  The  capabilities 
of  this  netwoik  have  not  been  tested  over  multiple  days  of  images  or  for  larger  data  bases  and  it  is 
believed  that  these  tests  will  cause  problems  for  the  network. 

2.4.2  Backpn^pagation  Neural  Network  for  Face  Verificatioii.  Researchers  at  die  Los 
Alamos  National  Laboratory  have  developed  a  face  verification  system  diat  uses  a  3-layer  neural 
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irigure  2.7.  Holons  derived  by  PCA  from  hidden  unit  responses.  (5) 


network  trained  by  back  propagation  (25).  The  netwt^  parameters,  which  were  arrived  at  eiqrirically, 
are  shown  below: 


•  Input  nodes:  1400 

•  Hidden  nodes:  20,  sigmoid  activation  range  [0,1] 

•  Output  nodes:  1,  sigmoid  activation  range  [0,1] 

•  Momentum:  0.50 

•  Hidden  layer  learning:  0.15 

•  Output  layer  learning:  0.30 

The  netw<^  is  fully  connected  and  all  wei^ts  are  initially  set  to  random  values  uniformly 
distributed  between  —0.1  and  0.1.  The  input  vectors  are  scaled  to  the  range  [-1,1]  and  the  network 
was  trained  to  ouqnit  a  1.0  for  the  target  and  aO.O  for  ou^ts  other  dian  die  target  The  training  was 
su^iped  after  1 5,000  iterations.  In  all  cases  die  nuitdier  of  target  pictures  in  the  data  base  was  artificially 
forced  to  be  10  percent  of  die  total  by  replicating  die  target  images. 
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The  data  base  used  contains  11,416  images  of  760  different  people  with  the  nuniber  of  images 
per  person  ranging  from  S  to  20.  All  images  were  aquired  using  a  video  camera  widi  VideoPix  installed 
on  a  Sun  Microsystems  SpaicStadon  IPC  computer.  As  can  be  seen  in  Hgure  2.8,  the  distance  from 
the  camera,  background,  and  lighting  conditions  were  kept  as  constant  as  possible. 


Figure  2.8.  Setup  used  for  capturing  images  at  Los  Alamos  National  Laboratmy.  Images  are  then 
used  in  a  Backprc^ngation  Netwok.  (25) 

The  netwmk  was  tested  using  various  compositions  of  training  data.  The  training  sets  consisted 
of  5,  10,  15,  OT  20  percent  of  the  data  base  either  randomly  selected  or  from  specific  demogra[Mc 
groups.  The  results  of  the  verification  tests  were  avoaged  over  all  training  scenarios  and  Ok  numbos 
show  a  99.997  percent  coirect  rejections  of  non-taigets  and  a  91.3  percent  correct  acceptances  of 
targets.  As  in  the  Cottrell  tests,  die  data  used  for  this  network  testing  was  captured  during  a  single 
sitting  on  one  day.  To  be  useful,  a  verification  network  must  perform  well  mi  multiple  days  of  data 
and  this  network  is  not  expected  to  perform  well  on  multiple  da3rs.  Additionally,  tiiis  network  has  not 
been  tested  for  classes  of  data  larger  tiian  one  so  die  upper  limits  of  die  network  are  unknown.  These 
concerns  will  be  addressed  in  this  research. 
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2.4.3  Additional  Network  Research  An  image  recognition  system  whidi  uses  an  infrared 
illumination  system  to  overcome  ambient  illumination  is  currently  being  developed  in  Fngland  (10). 
The  system  is  based  on  a  neural  netwcuk  known  as  WISARD  and  is  reported  to  be  cqMble  of  face 
recognition  against  background  scenes.  The  drawback  of  this  system  is  the  coiq>leuty:  it  requires  two 
cameras,  control  hardware,  an  IR  Illumination  System,  Training  Software,  and  WISARD  hardware. 
Likewise,  it  requires  a  significant  amount  of  meiiK^  because  two  800x540  images  are  taken  for  every 
scene. 

In  addition  to  WISARD  based  system,  a  neural  network  for  face  recognition  based  on  a  multilayer 
perceptron  and  shared  weights  has  been  introduced  in  Ftance  (4).  It  is  believed  that  diis  sytem 
will  handle  changes  in  lighting  and  rotations  and  face  translations  by  adding  additional  prototypes, 
neliminary  test  results  have  yielded  acceptable  results  but  the  database  consisted  of  only  10  subjects. 
This  system  will  also  require  large  amounts  of  memory  because  the  images  used  were  256x256  pixels, 
with  70  prototypes  captured  per  person.  This  equates  to  tqqnoximately  640  Megabytes  of  data. 

2.5  Summary 

This  chapter  {Resented  several  neural  network  a|>{>roaches  to  feature  extraction,  data  conqxes- 
sion,  and  classification.  Specifically,  in  relation  to  face  recognition,  an  identity  network,  a  Cottrell 
classification  netwmk  and  a  backprofngation  network  were  discussed.  Each  was  reported  to  perform 
well  for  face  recognition,  however,  the  data  bases  used  were  limited  in  size  and/or  in  die  nurhber  oS 
classes.  These  limitations  will  be  addressed  in  dus  research  by  increasing  the  size  of  die  «hifn  bases 
and  increasing  die  nuniber  of  classes  in  the  data  bases.  Additionally,  die  amiabilities  of  these  networks 
for  data  captured  over  multiple  days  has  not  been  investigated.  Therefore,  diis  research  will  also  focus 
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on  oqiNiiring  data  over  multipk  days  to  determine  bow  diis  type  of  data  effects  the  networks  described 
previously.  Chapter  3  presents  the  mediodology  used  to  address  dtese  issues. 
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III.  Methodology 


3.1  General 

The  objective  of  diis  thesis  was  to  investigate  various  neural  networks  for  face  recognition 
to  detemiine  their  c{4)abilities  as  feature  extractMS  and  classifiers.  Since  one  of  die  criteria  was 
to  inqilement  the  algorithms  on  Sun  woricstadons,  die  first  task  was  to  port  an  existing  multilayer 
perception  algcaithm  onto  a  Sun  woricstation  and  test  its  (^ration.  Next,  die  algoridim  was  enhanced 
to  w(Hk  in  die  following  modes  (refer  to  Af^ndix  A  for  code): 

•  As  a  multilayer  perception  (MLP)  with  sigmoidal  or  symetrical  sigmoidal  activations 

•  As  an  identity  netwrak  fcM*  image  compression  and  feature  extraction 

•  As  a  Cottrell  classification  networic 

•  As  a  4-layer  feature  extraction  and  classificmion  netwrak 

After  the  MLP  algorithm  was  ported  and  tested,  the  second  task  was  to  develop  and  test  the 
identity  network.  Next,  the  problem  of  individual  verification  was  examined  For  this  task,  the 
algorithim  were  used  to  verify  individuals  in  a  two-class  problem  where  class  1  signified  a  positive 
verification  and  class  2  an  unknown  or  negative  verification.  Next,  the  individual  verification  problem 
was  expanded  to  include  multiple  days.  The  multiple  day  problem  is  an  inqxHtant  extension  because  it 
has  not  been  investigated  in  the  literature  to  date.  After  verification  tests  were  complete,  die  algoriduns 
were  used  in  a  huge  scale  recognition  problem.  This  differed  from  die  verification  task  in  diat  die 
network  was  now  a  multiclass  problem  with  many  individuals  to  recognize  in  the  database  and  this 
is  a  much  more  difficult  problem  than  verification.  The  databases  used  in  both  die  verification  and 
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multiple  class  tests  were  conqiared  using  a  standard  multilayer  perceptron  and  the  neural  network 
feature  extractor/classifier.  Additionally,  the  performance  of  a  network  using  standard  Karliunen- 
Lofeve  Transform  (KLT)  coefficients  as  input  features  was  conq)ared  to  die  neural  network  extracted 
features.  The  final  task  was  to  describe  the  algorithms  develqied  to  inclement  the  above  objectives. 

3.2  Algorithm  Development 

The  first  phase  was  to  port  an  existing  MLP  algmithm  with  back  prt^iagation  learning  to  a  Sun 
workstation.  The  MLP  algorithm  used  for  this  {diase  allowed  a  maximum  of  three  layers  of  nodes 
(input,  hidden,  and  output)  and  used  sigmoidal  activations  in  the  hidden  layer  and  output  layer.  The 
sigmoids  in  this  case  had  limits  of  [0,1].  The  algrmdim  was  dien  modified  to  function  as  bodi  a  Cottrell 
and  an  Oja  identity  network.  For  this,  symmetrical  sigmoid  activation  functions,  limits  of  [-1,1],  were 
added  to  the  algorithm  and  were  interchangeable  widi  the  current  sigmoid  activation  functions.  The 
difference  between  the  Cottrell  identity  network  and  the  Oja  netwmk  is  diat  the  Oja  network  uses  linear 
activations  on  the  ou^ut  layer.  Additionally,  the  cipibility  to  display  the  network  input  image  and  the 
output  image  was  added  to  allow  for  visual  verifiouion  of  the  performance  of  the  identity  networks. 
The  next  enhancement  to  the  algotidun  was  to  allow  it  to  perform  as  a  C^ottrell  face  classification 
network.  This  meant  diat  the  network  algcnithm  had  to  initialize  and  learn  the  output  layer  weights  but 
the  input  layer  weights  were  those  previously  learned  from  an  identity  netwnk  algtHithm.  The  final 
enhatKement  to  the  algorithm  was  to  create  a  4-layer  network  fn*  feature  extraction  and  classification. 
This  was  done  by  adding  another  layer  to  the  Cottrell  classification  networic.  It  is  believed  that  this 
additional  layer  will  allow  for  increased  accuracy  in  the  classification  of  data  since  it  essentially  adds  a 
hidden  layer  to  the  classification  portion  of  the  netwtnk.  It  is  simply  a  backpropagation  network  whose 
inputs  are  the  outputs  of  the  hidden  layer  of  an  identity  network,  see  Figure  3.1. 


Output  Nodes 


OwUcMkiaNel: 


Extracted  Feaciires  are 
inputs  to  CUssifiotioa 
Network 


Input  Nodes  (Image  vector  pixel  values) 


Figure  3.1.  4-Layer  network  for  feature  extraction  and  classification.  The  input  layer  consists  of 
the  input  layer  of  a  previously  trained  identity  netwrak  and  the  upper  layers  consist  of  a 
multilayer  perception  with  backpn^iagation  learning. 


3.3  Feature  Extraction  and  Classification  Using  Identity  Networks 

The  goal  for  this  task  was  to  determine  whetlw  ch’  not  an  identity  netwcxk,  shown  in  Figure  3.2, 
could  extract  features  that  were  meaningful  as  infMits  to  a  classification  network.  The  idea  was  to  allow 
die  identity  network  to  kam  a  con^ssed  representation  of  die  data  and  reconstruct  it 

The  data  for  this  task  and  subsequent  tasks  consisted  of 32x32  ( 1 024)  pixel,  8  bit  gray  scale  images 
of  faces  that  had  been  ciqitured  on  die  Sun  workstations  using  VideoPix.  Each  individual  had  between 
10  and  20  images  captured  on  any  given  day  with  a  total  of  800  images  of  49  different  individuals.  All 
images  were  windowed  with  a  gaussian  window  routine  to  deemphasize  die  background. 
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The  netwoik  required  for  feature  extraction  consisted  of  1024  input  nodes  and  1024  ouq>ut 
nodes.  The  number  of  hidden  nodes  used  determined  die  amount  of  conqiression,  or  the  number  of 
features  extracted,  and  this  number  was  varied  from  10,  to  20,  to  40.  40  was  used  as  an  upper  limit 
due  to  training  time  required.  The  network  was  trained  until  the  averaged  mean  squared  error  per  pixel 
(MSE/P)  between  the  original  image  and  the  recb.,structed  image  was  less  than  .0034  which  corresponds 
to  25  gray  scale  levels  per  pixel  or  10  percent  reconstmction  error  per  pixel.  The  reconstruction  in 
this  manner  is  recognizeable  to  a  human  as  will  be  shown  in  ch^iter  4.  MSE/P  was  found  using  the 
following  definition 


MSE/P  = 


iM){N) 


(3.1) 


where  A/  is  the  image  size  in  pixels,  Af  is  the  number  of  images,  and  IN  and  OUT  are,  respectively, 
the  input  and  ouqiut  node  values. 


Input  Image 


Identity  Net 


Output  Image 


Hidden  node  outputs 
are  extracted  features 


Figure  3.2.  Image  compression/feature  extraction  and  reconstruction  procedure:  The  outputs  of  die 
hidden  nodes  are  the  features  used  as  inputs  to  a  classification  netwtnlc.  (1 1 , 5). 
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In  addition  to  varying  the  number  of  hidden  nodes,  the  size  of  the  database  in  terms  of  the  number 
of  classes  was  also  varied  to  determine  what  effects,  if  any,  this  had  on  the  netwoilcs.  The  databases 
used  for  these  tests  were  varied  from  10  classes,  to  30  classes,  to  49  classes. 

3.4  Verification  and  Classification  Using  Backpropagation 

This  task  is  divided  into  two  phases:  single  day  verification  and  multiple  day  verification.  The 
two  differ  in  the  data  sets  used  for  training  and  testing  die  networic.  Single  day  data  ate  images  that 
were  captured  on  one  day  only  within  a  90  second  window  for  each  individual.  Multiple  day  data  are 
images  that  were  captured  over  several  days  for  each  individual. 

3.4. 1  Single  Day  For  this  task  several  databases  were  generated  that  had  the  target  individual 
to  verify  as  class  1  and  images  of  other  subjects  as  class  2.  Multiple  runs  were  made  of  each  database 
and  the  average  accuracy  (correct  verification)  was  recorded.  The  minimum  number  of  images  used 
in  any  database  was  100  to  keep  in  line  with  the  testing  performed  in  (25).  The  mix  of  individuals  in 
each  data  set  was  20/80,  which  means  that  20  percent  of  the  images  in  the  data  set  were  of  the  target 
individual  (class  1)  and  80  percent  were  of  nontargets  (class  2).  Nontargets  consisted  of  2  images  each 
of  other  individuals  in  the  data  base. 

The  total  number  of  images  in  each  database  was  50:  10  fcH-  the  target  (class  1 )  and  40  (2  images 
each  of  20  other  individuals)  for  the  non-target  (class  2).  Each  database  was  trained  and  tested  using 
the  following: 

•  A  standard  multilayer  perceptron  with  nonlinear  activations  on  the  ouQrut  layer  and  hidden  la^r, 

•  A  Cottrell  (single  layer)  classification  network  with  Cottrell  identity  network  extracted  features 

as  inputs. 
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•  The  4-layer  network  which  uses  identity  networic  extracted  features  as  inputs  to  a  classification 
networic  with  a  hidden  layer. 

This  totaled  6  tests  for  each  database.  There  were  also  additional  tests  to  detennine  the  numbers  of 
hidden  nodes  and  iterations  which  provided  for  the  best  performance. 

3.4.2  Multiple  Day  Verification  This  task  is  one  of  the  nwst  significant  contributions  of  this 
thesis  because  it  has  not  received  attention  in  the  face  recognition  literature.  It  is  the  verification  of 
individuals  using  databases  that  are  generated  over  time.  Most  research  consists  of  images  tiiat  have 
been  c{q)tured  in  a  relative  short  period  of  time,  one  or  two  minutes,  and  under  very  strict  conditions. 
This  task  was  developed  to  test  the  system  over  less  constrained  input  images  because,  in  reality,  we 
change  fiom  day  to  day.  The  questions  are  then,  how  does  data  from  multiple  days  effect  the  accuracy 
and  what  are  the  limitations  of  this  system?  The  tq^proach  was  as  follows: 

1 .  Use  the  databases  from  the  individual  verification  task  to  train  the  various  netwcaks. 

2.  Test  each  network  using  data  captured  from  a  new  day. 

3.  After  testing,  add  the  new  days  data  to  the  training  database  and  retrain  the  netwoik. 

4.  Go  to  step  2,  and  continue  this  process  for  the  desired  number  of  days. 

3.4.3  Multiple  Person  Recognition  Fot  this  experiment,  the  size  of  the  database  was  in¬ 
creased  as  well  as  the  number  of  classes.  This  is  a  more  difficult  problem  tiian  individual  verification 
because  we  are  now  attempting  to  classify  (recognize)  many  faces  as  opposed  to  just  one  face.  The 
database  for  this  task  consisted  of  800  images  of  49  different  individuals.  Tests  were  accon^Ushed 
on  multiple  class  problems  using  10  classes  (10  different  people),  30  classes,  and  49  classes.  As  in 
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earlier  tests,  each  individual  had  10  or  20  images  cqitured  and  the  data  was  randomly  separated  in  each 
case  to  be  60  percent  training  data  and  40  percent  test  data.  The  point  of  this  part  of  the  study  was  to 
determine  what  effects  larger  databases  have  on  the  networlcs  being  tested. 

Several  questions  to  be  answered  are  listed  below. 

•  What  is  the  trade-off  between  the  size  of  the  database  and  classification  accuracy? 

•  How  many  hidden  nodes  ate  requited  for  a  given  number  of  classes? 

•  Do  larger  databases  make  training  time  prohibitive? 

3.5  Neural  Network  Classification  versos  Karhunen-Loive  Tiransfonn 

For  this  part  of  the  study  the  classification  accuracy  using  the  identity  network  extracted  fea¬ 
tures  was  conqrared  to  classification  accuracy  using  standard  Karhunen-Loive  Transform  fieatures 
(coefficients).  The  results  for  multiple  person  recognition  using  the  Karhunen-Loive  Transfcxm  ate 
documented  in  a  previous  thesis  by  Pedro  Suarez  (36)  and  in  a  collateral  thesis  by  Kennetii  Runyon 
(30).  Suarez  showed  that  the  Karhunen-Lo&ve  Transform,  using  single  day  data,  obtained  an  accuracy 
of  95  percent  on  55  faces  (classes  of  data).  Runyon  used  the  Kathunen-Loftve  Transfcwm  on  data 
gathered  using  a  motion  based  segmentation  system  developed  in  a  thesis  by  Kevin  Gay  (13).  Runyon 
had  data  sets  for  single  day,  multiple  classes  and  two  day,  multiple  classes.  Using  tire  motion  segmented 
data  and  the  Karhunen-Loive  Transform  Runyon  achieved  accuracies  of  77  percent  for  single  day  and 
32  percent  for  two  day  data  using  23  faces  (classes  of  data).  The  testing  in  tiiis  section  will  be  for 
a  comparison  of  the  Suarez  and  Runyon  systems  u>  the  neural  network  system.  As  sudi,  data  for  a 
multiple  class,  single  day  training  and  multiple  class,  two  day  training  will  be  used  for  comparisons. 
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Additionally,  the  data  sets  used  for  single  and  two  day  testing  in  die  neural  networks  will  be  tested  in 
the  Runyon  system. 

3.6  Code  Development 

All  code  for  this  thesis  was  written  in  ANSI  standard  C.  Some  routines  were  develtqied  specif¬ 
ically  for  this  thesis,  some  were  borrowed  from  others  and  some  were  taken  from  Numerical  Recipes 
In  C  (26).  Code  developed  for  this  thesis  is  included  in  Appendix  B. 

3.7  Summary 

This  chapter  presented  the  methodology  for  investigating  various  neural  networks  used  fw 
feature  extraction  and  classification  of  face  image  data.  Specifically,  an  identity  network  was  develqied 
and  tested  as  a  nonlinear  neural  network  feature  extractor.  Likewise,  a  single  layer  (Ckittrell)  and  a 
multilayer  backpropagation  networic  were  develcqied  for  use  as  classification  networks  widi  identity 
netwmk  extracted  features  as  inputs.  Additionally,  a  backprqiagation  netwtnlc  for  verification  and 
recognition,  using  data  gathered  over  multiple  days,  was  developed  and  tested.  In  all  cases,  data  bases 
used  for  testing  and  training  were  varied  by  the  number  of  exemplars  used,  by  the  number  of  classes, 
and  by  the  days  on  which  the  data  was  gathered.  Test  results  for  these  networks  are  found  in  Chapter  4. 
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IV.  Results 


4.1  General 

This  clu^r  presents  the  test  results  for  the  tasks  outlined  in  Qu4)ter  3.  Images  used  were  32x32, 
8  bit  gray  scale  that  were  captured  using  a  Sun  workstation  tool  called  VideoPix.  Mote  information  on 
VideoPix  can  be  found  in  a  collateral  thesis  (13).  Images  were  ptei»ocessed  to  add  a  gaussian  wiiulow 
around  the  faces  in  order  to  de-en^hasize  the  background. 

4.2  Feature  Extraction  and  Classification  Using  Identity  Networks 

4.2.1  Feature  Extraction  The  itiuige  comiHession  ciq)abilities  of  the  identity  netwtxk  was 
tested  first  because  the  ou^uts  of  the  hidden  layers  in  such  a  network  will  be  the  extracted  features  for 
use  in  a  classification  network.  The  networic  used,  shown  in  Hgure  2.4,  had  the  following  parameters 
which  were  based  on  the  work  of  Cottrell  (S)  and  Cja  (23): 

•  Input  nodes:  1024 

•  Hidden  nodes:  10, 20,  or  40,  sigtiwid  activation  range  [-1,1] 

•  Output  nodes:  1024,  sigmoid  activation  range  [-1,1] 

•  Momentum:  0. 

•  Hidden  layer  learning:  0.0001 

•  Ou^t  layer  learning:  0.1 

Training  the  netwrak  using  these  parameters,  especially  as  die  number  of  classes  increased, 
caused  the  ouQiut  error  to  begin  bouncing  slighdy,  see  Bgure  4.1,  indicating  that  the  learning  rate  used 
could  be  too  large. 
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Figure  4. 1 .  Non-smooth  identity  network  learning.  Plot  B  represents  a  more  detailed  look  at  Plot  A. 

Non-smooth  learning,  refer  to  Plot  B,  can  be  seen  after  4000  iterations,  indicating  that 
the  learning  rate  is  too  large.  In  this  case  eta  =  0. 1 


The  solution  was  to  retrain  the  network  using  a  smaller  output  layer  learning  rate;  0.01  instead  of 
the  0.1  of  the  original  training.  This  solved  the  non-smooth  behavira’,  see  Hgure  4.2,  but  the  final  MSE 
was  now  generally  above  the  target  range  of  3.5  after  the  desired  number  of  iterations.  This  problem 
was  solved  using  the  following  variable  learning  rate:  0.25  for  epoch  number  1, 0.1  fnr  epochs  2-2S, 
and  0.01  for  epochs  26  and  higher.  The  increments  of  25  epochs  were  chosen  through  trial  and  error. 
This  variable  learning  provided  for  a  smoother  MSE  curve,  see  Figure  4.3,  and  limited  the  training 
time  of  each  network,  run  on  Sun  SPARCstation  2’s,  to  8  hours  or  less. 

With  the  parameters  now  defined,  the  identity  network  perftMmed  as  desired  and  the  output  of 
the  network  at  various  stages  of  learning  is  shown  in  Figure  4.4.  As  evidenced  in  the  figure,  the 
network  can  learn  to  cort^rress  and  reconstruct,  to  an  acceptable  level,  the  images  in  a  data  set  Results 
of  testing  the  network  using  various  class  sizes  and  hidden  nodes  is  shown  in  Table  4.1. 

The  table  lists  the  final  average  MSE  for  all  images  in  the  training  set  after  15, 000  iterations. 
The  results  show  that  the  target  MSE  of  3.5  or  less  was  achieved  for  10  and  30  classes  but  was  slightly 
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Hguie  4.2.  Slow  identity  networic  learning  indicating  that  the  learning  rate  is  too  small.  In  this  case 
eta  =  0.01 


Table  4.1.  Average  MSE  of  an  Identity  Netw<»k  after  15000  iterations  for  varying  class  sizes  and 
numbers  of  hidden  layer  nodes.  For  die  Identity  Network,  inputs  =  outputs  =  1024. 


10  nodes 

20  nodes 

40  nodes 

10  classes 

2.7 

1.6 

0.9 

30  classes 

3.3 

2.3 

2.1 

49  classes 

5.3 

4.0 

7.3 

higher  for  49  classes.  The  15, 000  iteration  limit  was  placed  on  die  networks  to  limit  the  training  time 
and  it  is  believed  that  the  final  error  for  49  classes  could  be  lowered  given  a  mudi  longer  training  time: 
this  was  tested  with  a  single  run  where  a  final  MSE  of  3.8  was  readied  after  50, 000  itnations. 

Finally,  it  should  be  noted  that  the  input  layer  weights  learned  by  each  identity  networit  were 
saved  ftn’  use  as  the  input  layers  of  the  classification  networks  because  the  output  activations  of  this 
layer  would  be  the  features  used  in  the  classification  netwraks. 

4.2.2  Classification  Using  Identity  Network  Extracted  Features  i^th  die  identity  networks 
trained,  die  next  phase  was  to  test  how  well  feiuures  extracted  from  the  identity  networks  worked  in 
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Figure  4.3.  Identity  network  kaming  obtained  with  variable  learning  rate.  The  net  learns  quickly  at 
first  and  more  slowly  and  smoothly  as  the  iterations  increase.  Plot  B  represents  a  more 
detailed  look  at  Plot  A. 


a  classification  network.  Recall  that  the  extracted  features  used  here  are  the  output  activations  of  tiie 
hidden  layer  nodes  of  an  identity  network,  refer  to  Figure  3.2.  For  diis  test,  tiie  network  used  is  tiiat 
shown  in  Hgure  3. 1  where  the  input  layer  is  the  fixed  weights  from  the  previously  trained  identity 
netwoiks  and  the  output  layers  are  a  backpropagation  networic  with  die  following  parameters. 


•  Input  nodes:  1024 

•  Hidden  layer  1  nodes:  Fixed  by  Identity  net  training  at  10, 20,  or  40 

•  Hidden  layer  2  nodes:  Varied  from  0,  for  a  Cottrell  classifier,  to  50 

•  Output  nodes:  Multiple  classes  of  10, 30,  or  49 

•  Momentum:  0.50 

•  Hidden  layer  learning:  0.15 

•  Output  layer  learning:  0.30 
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Figure  4.4.  Identity  netw<^  input  images  versus  reconstructed  output  images  at  selected  iterations 
of  0.  3000. 6000. 9000. 12000.  and  15000. 


•  Activation  Function:  Symmetrical  sigmoid,  range  ['I,!],  at  hidden  layer  1  and  sigmoid  with 
range  [0,1]  at  hidden  layer  2  and  the  output 

The  results  of  the  various  classification  network  configurations  are  summarized  in  Tables  4.2 
and  4.3,  which  show  that  the  features  extracted  from  die  identity  netwoiks  are  acceptable  features  to 
classify  the  data. 

The  results  show  that  the  feature  extraction  and  classification  netwoiks  used  by  Cottrell  (a  100/1 
conquession  or  10  hidden  nodes  in  identity  network  and  no  hidden  layers  in  the  classification  network) 
had  a  significant  decrease  in  classification  accuracy  as  the  number  of  classes  was  inoeased.  However, 
die  accuracy  was  improved  by  decreasing  the  Cottrell  recommended  conqiression  of  lOQ/1  to  25/1  (40 
hidden  nodes  in  the  identity  nets).  Additionally,  adding  a  hidden  layer  of  nodes  to  the  classifier  also 
improved  die  classification  accuracy  of  die  test  set,  refer  to  Table  4.3.  As  die  table  shows,  varying 
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Table  4.2.  Classification  Accuracy  Using  Identity  Netwock  Extracted  Features  as  inputs  to  a  Cottrell 
(no  hidden  layer)  baclq>ropagationnetw«k.  As  the  class  size  increases  die  Cottrell  network 
using  10  input  features  performs  poorly.  The  accuracy  was  greatly  improved  when  die 
Cottrell  network  was  modified  to  have  20  and  40  input  features. 


Coorell  Classifier  Test  Set  Accuracies 

Input  Features 

CXitput  Classes 

Test  set  accuracy 

10 

10 

87.5% 

10 

30 

85.0% 

10 

49 

66.0% 

20 

10 

95.0% 

20 

30 

97.0% 

20 

49 

91.8% 

40 

10 

85.0% 

40 

30 

96.7% 

40 

49 

97.9% 

the  number  of  hidden  nodes  (between  10  and  SO)  in  die  classification  portion  of  die  network  had  a 
slight  impact  on  the  test  set  accuracy;  too  few  or  too  many  nodes  in  die  hidden  layer  caused  the  test 
set  accuracy  to  decrease.  The  trade  off  then  is  between  the  accuracy  desired  and  the  training  time 
involved  fcx'  larger  networks.  As  stated  earlier,  the  identity  nets  with  40  hidden  nodes  required  8  hours 
of  training  time  on  Sun  workstations;  this  must  be  added  to  the  time  to  train  die  classification  portion 
of  each  netwmk,  and  this  time  grows  as  die  number  of  hidden  nodes  is  increased.  However,  given  die 
trade  offs,  die  bottom  line  here  is  diat  the  features  extracted  using  the  identity  networks  can  be  used  to 
classify  die  data  with  acceptable  accuracy. 

When  comparing  these  results  to  those  obtained  by  Cottrell  (11,  S),  a  KXVl  compression  ratio 
(10  hidden  nodes)  did  not  yield  die  99  percent  accuracies  he  reported.  In  fact,  as  the  nundier  of  classes 
was  increased  to  49,  which  is  over  twice  the  20  classes  Cottrell  used,  die  accuracy  drops  to  a  low  of  only 
66  percent  However,  by  increasing  the  mmibtt  of  hidden  nodes  to  40,  whidi  increases  the  nuiidier 
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Table  4.3.  Gassificadon  Accuracy  for  test  data  using  Identity  Network  Extracted  Features  as  inputs 
to  a  single  hidden  layer  baclq>ropagation  network.  This  type  of  network  is  an  improvement 
over  the  Cottrell  classification  netwcMk.  Accuracies  are  listed  with  respect  to  the  number 
of  nodes  in  the  hidden  layer.  All  training  sets  obtained  a  100%  training  accuracy  during 
training. 


Input  Features 

OuQMit  Classes 

10  nodes 

20  nodes 

30  nodes 

40  nodes 

50  nodes 

10 

10 

90.0% 

90.0% 

97.5% 

92.5% 

90.0% 

10 

30 

80.8% 

85.0% 

89.1% 

85.0% 

85.0% 

10 

49 

64.3% 

71.4% 

73.0% 

73.0% 

75.0% 

20 

10 

95.0% 

95.0% 

95.0% 

97.5% 

95.0% 

20 

30 

88.3% 

91.6% 

93.3% 

94.1% 

89.1% 

20 

49 

77.0% 

84.1% 

89.7% 

88.2% 

86.2% 

40 

10 

85.0% 

95.0% 

85.0% 

85.0% 

80.0% 

40 

30 

92.5% 

94.1% 

96.7% 

93.3% 

91.7% 

40 

49 

80.0% 

91.8% 

91.8% 

91.3% 

92.8% 

of  features  (inputs)  used  in  the  classification  network,  the  classification  accuracy  increased  over  30 
percent  Therefore,  modifying  the  Cottrell  identity  netwwk  to  a  25/1  cotiqiression  ratio  allows  the 
network  to  perform  well  for  class  sizes  as  larger  as  49.  It  is  also  important  to  note  that  for  this  research 
the  input  images  were  32x32  pixels  and  Cottrell  used  images  of  128x128  inxels;  in  other  words,  die 
modified  identity  network  (ie.  25/1  compression)  perftxms  well  and  allows  a  75  percent  reduction  in 
the  amount  of  input  data  required. 

4.2.3  Identity  Networks  and  Multiple  Day  Data  Classiflcation  For  tiiis  test,  data  gatiiered 
on  a  second  day  was  used  to  test  the  identity  networks  generalization  capabilities  for  feature  extraction. 
Specifically,  the  data  from  day  2  was  input  to  die  previously  trained  identity  networks  (trained  using  day 
1  data)  and  the  features  for  that  data  (activations  of  the  hidden  nodes)  were  saved.  The  features  were 
then  tested  using  the  weights  of  the  previously  trained  classification  networks  of  Table  4.3.  Results  of 
diis  testing  are  shown  in  Table  4.4.  These  results  show  that  the  identity  networks  do  not  generalize 
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well  over  multiple  days  of  data.  Likewise,  adding  additional  days  worth  of  data  to  the  data  sets  did 
not  inqnrove  the  classification  accuracy.  This  is  because  multiple  days  of  Ham  create  problems  for  die 
identity  networks;  the  identity  network  can  minimize  the  MSE  of  die  training  set  but  when  multiple 
days  of  data  are  used,  the  error  on  the  test  set  remains  a  magnitude  higher  which  means  that  the  test  set 
error,  in  terms  of  reconstruction,  is  plus  or  minus  100  percent  As  such,  die  features  extracted  by  die 
identity  network  for  the  test  set  are  not  adequate  for  proper  classification. 

Table  4.4.  Classification  Accuracy  using  Identity  Network  Extracted  Features  of  multiple  day  data  as 
inputs  to  a  2  weight  layer  backpropagation  netwMk  for  varying  numbers  of  hidden  layer 
nodes  and  a  single  weight  layer  Cottrell  classification  network.  NOTE:  0  nodes  represents 
the  Cottrell  network  results. 


Output  Classes 

Input  Features 

10  nodes 

20  nodes 

30  nodes 

40  nodes 

50  nodes 

0  nodes 

10 

10 

5.0% 

5.0% 

12.0% 

10.0% 

3.0% 

10.0% 

10 

20 

5.0% 

8.0% 

8.0% 

10.0% 

5.0% 

10.0% 

10 

40 

10.0% 

19.0% 

19.0% 

20.0% 

14.0% 

10.0% 

30 

10 

5.0% 

5.0% 

3.0% 

1.0% 

3.3% 

3.3% 

30 

20 

5.0% 

6.7% 

2.3% 

4.3% 

4.7% 

3.3% 

30 

40 

6.0% 

4.7% 

5.3% 

5.3% 

3.3% 

3.3% 

4.3  Veriflcation  and  Classificatioa  Using  Backpropagation 

4.3.1  Single  Day  Verification  The  images  used  for  this  test  had  been  captured  during  a  single 
sitting  of  each  subject  over  about  90  seconds.  The  training  and  test  data  sets  were  genoated  such  that 
the  target  subject  (for  verification)  was  class  1  and  all  other  faces  in  the  data  sets  were  class  2.  The 
network  used  for  this  test  had  the  following  parameters: 

•  Input  nodes:  1024 

•  Hidden  layer  nodes:  10,  or  20 
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•  Output  nodes:  2 


•  Momentum:  0.50 

•  Hidden  layer  learning:  0.15 

•  OuQMit  layer  learning:  0.30 

•  Activation  Functions:  Sigmoidal  (range  [0.1])  at  hidden  and  output  layers. 

The  goal  was  to  affirm  that  that  the  netwtxk  could  craiectly  verily  a  target  individual.  Test 
results,  representing  10  runs  each  for  3  targets,  are  shown  in  Table  4.5.  The  results  show  diat  the 
networic  performed  as  expected  and  can  verify  individual  targets  using  a  standard  backpropagation 
network. 

Table  4.5.  Classification  Accuracies  fw  Single  day  verification  of  3  target  subjects 


Data  (day) 

Hidden  laya  nodes 

30  tun  average 

day  1  vs  day  1 

10 

96.4% 

day  1  vs  day  1 

20 

%.9% 

These  results  support  the  verification  work  being  performed  in  Los  Alamos  National  Laboratory 
(25)  which  uses  raw  image  data  as  inputs  to  a  backinxq)agation  netwtnk  with  a  single  hidden  layer.  The 
Los  Alamos  Laboratory  research  uses  a  minimum  of  100  images  to  train  the  network,  widi  10  patent 
of  die  images  representing  the  target  to  verify  and  90  percent  representing  nontargets.  Additionally, 
their  data  was  ciqitured  on  a  single  day  for  each  individual.  Data  captured  in  this  manna  yields  good 
results  (Los  Alamos  had  results  as  high  as  99  percent  for  some  individuals)  but  it  does  not  allow  for 
daily  changes  in  individuals  and,  therefoc,  does  not  represent  the  real  world  where  images  will  have  to 
be  oqMured  on  many  days  if  the  system  is  to  be  useable  for  verificaticm.  The  multiple  day  day  problem 
is  addressed  in  the  next  section. 
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4.3.2  Multiple  Day  Verification  The  accuracy  for  verification  using  data  captured  on  a  single 
day  is  very  good.  Howrever,  it  is  believed  that  training  on  a  single  days  data  will  not  allow  for  verification 
in  a  real  environment  where  images  will  be  gathered  daily.  This  sections  testing  was  designed  to  find 
the  problems  of  a  multiple  day  system  and  propose  and  test  solutions.  The  network  used  for  testing 
in  this  section  is  the  same  as  that  for  single  day  verification,  ie.  single  hidden  layer  baclqnopagation 
network,  but  the  difference  is  in  the  data  sets  used  for  training  and  testing.  Data  used  in  this  section 
was  gathered  over  several  days  with  multiple  sittings  of  each  target.  The  results  of  the  testing  are  found 
in  Table  4.6. 


Table  4.6.  Classification  Accuracies  for  Multiple  day  verification  of  3  target  subjects  using  raw  image 
data  as  inputs  to  a  single  hidden  layer  baclquopagation  network.  All  training  sets  obtained 
a  100%  classification  accuracy  during  training. 


Data  (day) 

Hidden  layer  nodes 

30  run  average 

day  1  vs  day  1 

10 

97.0% 

day  1  vs  day  1 

20 

96.6% 

day  1  vs  day  1 

40 

95.3% 

day  1  vs  day  2 

10 

11.7% 

day  1  vs  day  2 

20 

15.0% 

day  1  vs  day  2 

40 

20.5% 

day  1-2  vs  1-2 

10 

91.1% 

day  1-2  vs  1-2 

20 

90.8% 

day  1-2  vs  1-2 

40 

89.0% 

day  1-2  vs  3 

10 

9.0% 

day  1-2  vs  3 

20 

15.0% 

day  1-2  vs  3 

40 

2.0% 

day  1-3  vs  1-3 

10 

90.6% 

day  1-3  vs  1-3 

20 

90.7% 

day  1-3  vs  1-3 

40 

90.3% 

day  1-3  vs  4 

10 

14.0% 

day  1-3  vs  4 

20 

33.0% 

day  1-3  vs  4 

40 

44.0% 
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Column  one  indicates  which  day(s)  the  training  versus  test  sets  were  captured  on.  Column 
three  shows  the  classification  accuracies  for  the  various  data  sets  and  hidden  node  configurations.  As 
expected,  the  network  performed  poorly  in  the  multiple  day  test,  but  it  was  proposed  that  increasing 
the  number  of  days  in  the  training  set  (ie.  more  prototypes)  would  show  an  improvement  However,  as 
can  be  seen  in  the  table,  adding  prototypes  from  several  days  provided  no  improvement  over  the  single 
day  training.  Therefore,  a  test  was  developed  to  determine  why  the  netwwk  was  not  inqrroving  even 
with  the  introduction  of  multiple  days  images  into  the  training  set 

For  this  test  a  training  data  set  was  created  that  contained  target  Images  from  a  single  day  only 
and  the  networic  was  trained.  Then  a  test  set  was  created  by  shifting  these  images  1  pixel  at  a  time  and 
testing  the  net  to  determine  how  a  shift  in  the  image  would  effect  the  verification  accuracy.  Likewise, 
the  images  were  scaled  to  determine  if  scale  was  a  problem.  An  original  image  with  a  shifted  and 
scaled  version  are  shown  in  Figure  4.S  and  the  results  of  the  testing  are  provided  in  Table  4.7.  The 
results  of  these  test  indicate  that  shifts  of  1  pixel  can  be  overcome  by  some  network  configurations 
since  the  accuracy  for  a  1  pixel  shift  varied  from  0  to  100%.  However,  if  the  shift  is  2  pixels,  or  about 
a  6.6%  shift,  then  the  network  performs  poorly.  Likewise,  a  scale  change  of  even  5  percent  will  cause 
severe  problems  for  the  network. 

Table  4.7.  Classification  accuracies  of  original  (day  1}  images  versus  scaled  and  shifted  versions  of 
the  same  images.  Original  images  were  trained  to  100%  accuracy  of  the  training  set; 
original  image  test  set  accuracies  are  shown  in  the  table. 


Test  Set  Classification  Accuracy:  10  run  average,  10  test  images  per  nm 

Original  Images  ||  Shifted  1  pixel 

Shifted  2  pixels 

Scaled  S% 

99.0%  40.0% 

00.0% 

1  03.0% 
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Original  Scaled  Shifted 

Image  Image  Image 


Figure  4.5.  Original  image  and  a  scaled  and  shifted  version  of  the  same  for  testing  the  effects  of  scale 
and  shift  on  the  multiple  day  verification  networics 

The  question  was  then  how  do  you  compensate  for  the  scale  and  shift  problems?  More  {mitotypes 
still  seemed  to  be  a  piece  of  the  solution  but  that  alone  was  shown  previously  to  offer  little  improvement 
The  proposed  solution  was  to  increase  the  number  of  days  of  training  data  plus  increase  the  mixture  of 
target  versus  nontarget  images  in  the  training  set  This  means  that  instead  of  using  a  data  set  wiih  20 
percent  of  the  images  from  the  target  and  80  percent  nontarget  a  new  data  set  containing  a  SO/SO  mix 
would  be  created  and  tested.  It  was  believed  that  diis  would  allow  the  weights  to  update  mrae  evenly 
for  target  and  non  target  alike. 

The  networks  were  retrained  and  the  results  for  2  target  subjects  are  shown  in  Table  4.8.  As 
can  be  seen  in  the  table,  training  the  network  using  data  gathered  over  9  days  dramatically  increased 
the  classification  accuracy  for  multiple  day  test  sets;  in  this  case,  the  test  sets  consisted  of  data 
gathered  on  days  10  and  1 1 .  Therefore,  training  on  days  1  through  9  allowed  the  networks  to  correctly 
verify  target  data  from  days  10  and  1 1  with  a  high  degree  of  accuracy.  Given  the  above  results,  die 
solution  for  verification  over  multiple  days,  which  had  not  been  addressed  in  previous  literature,  is  a 
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Table  4.8.  Classification  Accuracies  for  Multiple  day  verification  using  data  sets  with  50%  taiget 
images  and  50%  nontaiget  images.  All  training  sets  attained  a  1 00%  classification  accuracy 
during  training. 


Data  (day) 

Hidden  layer  nodes 

30  run  average 

day  1-4  vs  1-4 

10 

88.8% 

day  1-4  vs  1-4 

20 

89.5% 

day  1-4  vs  1-4 

40 

88.3% 

day  1-4  vs  5 

10 

30.5% 

day  1-4  vs  5 

20 

11.0% 

day  1-4  vs  5 

40 

25.0% 

day  1-9  vs  1-9 

10 

91.4% 

day  1-9  vs  1-9 

20 

91.5% 

day  1-9  vs  1-9 

40 

89.7% 

day  1-9  vs  10 

10 

99.4% 

day  1-9  vs  10 

20 

100.0% 

day  1-9  vs  10 

40 

94.0% 

day  1-9  vs  10-1 1 

10 

94.7% 

day  1-9  vs  10-1 1 

20 

88.2% 

day  1-9  vs  10-11 

40 

86.5% 

baclq>ropagation  network  with  a  single  hidden  layer  and  a  training  set  with  an  equal  number  of  taiget 
and  nontaiget  data  images  collected  over  multiple  days. 

4.3.2. 1  False  Acceptance  Testing  This  task  was  designed  to  test  the  false  acceptance 
rate  for  nontaigets.  In  other  words,  what  percentage  of  the  time  will  the  netwoik  identify  a  nontaiget 
as  a  taiget  The  data  for  this  test  consisted  of  90  images  of  nontaigets,  gathered  over  several  days, 
and  they  were  tested  against  the  netwoiks  trained  fw  day  1-9  data.  Results  of  this  testing  are  shown 
in  Table  4.9.  The  table  shows  that  for  any  number  of  hidden  nodes  used  die  false  acceptance  was  no 
greater  dian  9.2  percent  As  stated  in  the  previous  section,  these  results,  and  the  verification  results  of 
Table  4.8,  indicate  diat  a  backpropagation  networit  with  a  single  hidden  layer  can  be  used  successfully 
for  verification  over  multiple  days. 
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Table  4.9.  False  acceptance  testing  of  multiple  day  images  of  nontaigets.  Data  was  tested  against 
networks  previously  trained  for  multiple  day  verification.  The  False  Acceptance  Rate 
indicates  how  often  a  nontaiget  was  identified  as  a  target,  based  on  a  20  run  average. 


Data  (day) 

Hidden  layer  nodes 

False  Acceptance  Rate 

day  1-9  vs  nontargets 

10 

6.9% 

day  1-9  vs  nontargets 

20 

9.6% 

day  1-9  vs  nontargets 

40 

8.2% 

4.3.3  Multiple  Person  Recognition  This  portion  of  the  testing  used  the  backprqwgation 
network  once  more  to  assess  the  ability  of  the  netwmk  to  classify  multiple  classes  of  face  images.  The 
parameters  for  the  network  remained  as  above.  An  immediate  disadvantage  of  using  die  larger  class 
networks  was  the  training  time.  As  a  minimum,  a  10  class  problem  with  40  hidden  nodes  required  eight 
hours  on  the  Sun  SPARCstations  to  train  and  as  the  number  of  classes  increased  so  did  die  training  time 
to  a  maximum’  time  for  these  tests  of  15  days  for  a  49  class  problem  with  250  hidden  nodes.  Results 
of  the  testing  for  this  section  are  shown  in  Table  4.10. 

For  single  day.  multiple  class  problems  die  network  could  learn  the  training  set  and  perform 
reasonable  well  on  the  test  set  accuracy,  but  when  a  second  days  data  was  added  to  die  training  and 
test  set  the  accuracy  was  again  a  problem.  No  further  testing  of  multiple  day,  multiple  class  could  be 
perfumed  to  determine  if  additional  prototypes  would  be  useful  because  data  for  more  dian  2  days 
only  existed  fOT  3  individuals,  all  other  subjects  were  2  days  of  data  only. 

’Training  to  200K  iterations,  which  took  13  days  and  the  training  data  only  learned  to  553%.  Estiniales  of  the  time 
required  to  complete  the  training  on  Sun  SPARCsrations  are  30  days. 
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Table  4.10.  Gassification  Accuracies  for  Multiple  Gasses  using  raw  image  data  (1024,  8-bit  gray 
scale  values)  as  inputs  to  a  single  hidden  layer  backpr(^>agation  networic.  All  training 
sets  attained  100%  classification  accuracy  during  training. 


Gasses 

Hidden  layer  nodes 

Iterations 

Test  Set  Accuracy 

10 

10 

lOK 

86.7% 

10 

20 

lOK 

91.6% 

10 

30 

lOK 

83.3% 

31 

31 

30K 

92.5% 

31 

62 

SOK 

92.9% 

31 

93 

50K 

81.7% 

49 

50 

lOOK 

91.9% 

49 

100 

lOOK 

87.1% 

49 

ISO 

200K 

74.6% 

49 

250 

200K 

46.4% 

4.4  Gassification  using  raw  data,  Karhunen-Lo^ve  transform  features,  and  identity  network 
extracted  features 

For  this  task,  multiple  classes  of  data  gathered  over  two  days  was  tested  in  the  neural  networics 
described  previously  and  in  the  AFTT  end-to-end  system  (developed  by  Runyon)  which  uses  the 
Kaihunen-Loive  Transform  (KLT)  as  developed  by  Suarez  (36).  The  purpose  of  diis  task  was  to 
compare  the  classification  accuracy  of  a  neural  network  using  three  different  input  features:  raw  ima^ 
data,  KLT  extracted  features,  and  identity  networic  extracted  features.  The  number  of  iiqmt  features 
and  hidden  nodes  used  were  determined  from  {xevious  tests — these  parameters  gave  the  best  results. 
The  data  sets  for  these  tasks  are  as  follows: 


•  Motion  Segmented  E)ata  (MSD).  This  consisted  of  230  images  of  23  subjects  (classes)  cq)tured 
over  two  days  with  the  motion  segmentation  system  described  in  (13). 

•  Hand  Segmented  Data  (HSD).  This  data  is  composed  of 600  images  of  30  subjects  ci^Kured  over 
two  days,  one  sitting  per  day,  with  the  system  described  in  charter  3. 
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The  identity  network  features  were  extracted  from  die  motion  segmented  data  and  the  hand 
segmented  data  and  the  reconstruction  of  images  using  diese  features  is  presented  in  Figure  4.6  fcv  the 
motion  segmentation  and  Figure  4.4  for  the  hand  segmentation.  Based  on  die  visible  reconstruction 
emx,  again  refer  to  Figure  4.6,  which  was  hypothesized  to  be  inqxMtant  to  some  degree  in  face 
recognition,  the  features  extracted  from  the  motion  segmented  data  should  not  do  as  well  as  die  hand 
segmented  data. 


Reconstruction  of  Motion  Segmented  Data 


InO 


0 


OutO 


In5K 


Out  5K 


InlOK 


Out  lOK 


Inl5K 


Out  15K 


In20K 


Out  20K 


Hgure  4.6.  Identity  netwoik  input  images  versus  reconstructed  output  images  using  Motitm  seg¬ 
mented  data.  The  images  are  shown  at  selected  itnations  of  0, 5000, 10000, 15000,  and 
20000.  Although  the  identity  network  reduced  die  avnage  MSE,  the  reconstruction  of 
this  data  is  not  recognizable  to  a  human  observer. 


The  test  results  are  shown  in  Table  4.11.  Gassification  accuracy  fw  hand  segmented,  single  day, 
multiple  classes  is  essentially  equivalent  for  all  types  of  inputs.  However,  using  stria  Karhunen-Lobve 
transform  features  as  inputs  to  a  classification  netwoik  yielded  higher  classification  accuracies,  by  a 
few  percentage  points,  than  did  the  identity  networic  extracted  feaures  as  inputs  to  die  classifier.  This 
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could  be  expected  since  the  identity  networics  develop  a  distributed  representation  of  the  principal 
components  and  not  necessarily  the  most  significant  principal  conqwnents  (1 1,  S)  as  does  the  strict 
KL  transform.  When  using  the  motion  segmented  data,  the  Kaihunen-Lobve  transform  features  were 
clearly  better  to  use  as  inputs  to  a  classification  netwrak.  When  combining  data  aq)tured  from  two 
days,  the  results  are  again  basically  equivalent  for  die  inputs  used.  Once  more,  however,  die  KLT 
extracted  features  gave  a  higher  classification  accuracy  when  using  the  motion  segmented  data.  These 
tests  clearly  indicate  that  the  KLT  extracted  features  as  iiqiuts  to  a  classification  netwoik  provide  the 
best  results.  The  motion  segmented  data  most  likely  gave  lower  results  for  the  KLT  features  because 
the  KLT  is  not  shift  or  scale  invariant  and  the  motion  segmented  data  is  much  more  susceptable  to  these 
variations. 

4.5  Code  Development 

As  stated  in  cluqiter  3,  all  code  was  developed  using  ANSI  Standard  C.  The  algorithms  developed 
as  part  of  diis  research  were  tested  using  carefully  developed  test  data  files.  Whenever  possible,  diese 
files  were  run  on  the  NeuralGraphics  (37)  system  and  code  develtqied  for  this  research  to  insure  proper 
operation  of  the  developed  code. 

4.6  Summary 

This  chapter  presented  the  results  of  testing  several  different  networks  used  for  recognition  or 
verification  of  fiace  images.  Networks  tested  consisted  of  a  back  prtqwgation  network  using  the  raw 
data  as  inputs;  a  back  propagation  network  using  Karfaunen-Lobve  Transform  coefficients,  computed 
from  the  raw  data,  as  inputs;  and  a  back  propagation  network  using  features  extracted  by  an  identity 
network  as  inputs.  As  discussed  in  (Thiqiter  2,  diese  networks  bad  not  been  tested  against  data  gathered 


4-17 


Table  4. 1 1 .  Gassification  Accuracies  for  raw  image  data  versus  Kaibuneo-Lobve  transfonn  extracted 
features  versus  identity  networic  extracted  features  as  inputs  to  a  back  propagatitHi  classi¬ 
fication  networic.  All  training  sets  attained  100%  classification  accuracy  during  training. 
For  this  table,  HSD  is  Hand  Segmented  Data  and  MSD  is  Motion  Segmented  Data 


Classification  Accuracies:  Raw  data  vs  KLT  coefficients  vs  Identity  network  features  as  inputs 


Dataset 

Network  Configuration  | 

Accuracy 

Inputs 

Hidden  Nodes 

Outputs  (classes) 

HSD  day  1 

1024  (raw  data) 

60 

30 

93.3% 

20(KLTcoeffs) 

40 

30 

97.0% 

40  (ID  features) 

40 

30 

93.3% 

HSD  dayl-i-2 

1024 

60 

30 

93.3% 

20 

40 

30 

9S.0% 

40 

40 

30 

93.8% 

HSDdaylvs2 

1024 

60 

30 

3.3% 

20 

40 

30 

S3.0% 

40 

40 

30 

S.3% 

MSDdayl 

1024 

60 

23 

37.0% 

20 

40 

23 

76.0% 

40 

40 

23 

42.4% 

MSDdayl-t-2 

1024 

60 

23 

40.0% 

20 

40 

23 

74.0% 

40 

40 

23 

41.0% 

MSD  daylvs2 

60 

23 

10.4% 

20 

40 

23 

34.0% 

40 

40 

23 

13.9% 

over  multiple  days.  As  such,  this  was  a  primary  focus  of  the  testing  during  this  effort  The  lesuUs  show 
diat  all  the  networics  perfonn  poorly  when  training  on  data  captured  on  a  single  day  and  testing  on  data 
gathered  on  a  different  day.  However,  die  most  important  result  fw  this  effwt  is  diat  verification  can  be 
accomplished  over  multiple  days  if  the  training  set  used  contains  data  gathered  over  many  days,  in  this 
case  9  days  was  sufficient,  and  if  die  training  set  is  composed  of  SO  percent  target  images  and  SO  percent 
nontarget  images.  For  this  composition  of  training  data,  using  a  single  hidden  layer  baclqiropagation 
network,  and  the  raw  image  data  as  inputs,  die  verification  accuracy  over  multiple  days  was  94  percent 
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and  the  false  acceptance  for  the  same  netw(»k  was  6.5  percent  This  indicates  that  face  voificadon 
over  multiple  days  can  be  performed  with  a  neural  network. 
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V  Condusioos 


5.1  General 

The  purpose  of  this  study  was  to  investigate  and  inclement  a  neural  network  for  face  verification 
and  classification.  The  objective  was  to  develop  a  neural  network  based  feature  extractor  and/or 
classifier  that  can  be  used  to  authorized  user  verification  in  a  realistic  work  environment  Specifically, 
tiuee  netwoiks  were  developed  and  tested:  a  back  prqjagation  network  using  the  raw  data  as  inputs; 
a  back  propagation  network  using  Kaihunen-Loive  Transform  coefficients,  computed  from  die  raw 
data,  as  inputs;  and  a  back  propagation  network  using  features  extracted  by  an  identity  network  as 
inputs.  The  following  sections  present  the  conclusions.  Hrst  the  multiple  day  problem  conclusion  is 
presented,  then  the  identity  networks  are  discussed.  This  is  followed  by  a  discussion  of  tire  multiple 
class  recognition  problem;  finally,  the  comparison  of  neural  network  extracted  features  to  the  strict 
Karhunen-Loive  Transform  features  is  presented. 

5.2  Multiple  Day  Vcrificatioa 

The  most  significant  conclusions  for  diis  research  are  based  on  the  multiple  day  verification 
results.  Multiple  day  verification  is  a  significant  problem  diat  had  not  been  addressed  previously  in  tire 
literature  and  the  objective  was  to  determine  if  a  neural  networit  could  be  used  to  solve  die  multiple 
day  verification  problem. 

The  solution  to  the  multiple  day  verification  problem,  supported  by  test  results  of  Chapter  4, 
is  a  single  hidden  layer  backpropagation  network  that  uses  taw  image  data  for  inputs.  When  diese 
networks  were  trained  as  two  class  networks  accuracies  of  100  percent  were  obtained  to  the  multiple 
day  data.  It  was  also  found  that  training  set  composition  is  important;  the  training  data  diat  provided 
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dte  best  results  was  a  data  set  conqwsed  of  SO  percent  target  individual,  captured  over  9  days,  and  SO 
percent  nontaiget  induviduals.  Using  less  than  9  days  wordi  of  data  proved  to  dramatically  deoease 
the  verification  accuracy  of  the  netwoiir.  This  is  most  likely  due  to  the  shift  and  scale  sensitivity  of 
these  netwtxks,  also  identified  during  testing.  Gadieting  9  days  of  images,  ie.  nuiltiple  prototypes, 
allowed  the  networic  to  learn  the  shifted  and  scaled  versions  of  die  target  which  dramatically  inovased 
the  verification  accuracies.  Again,  these  results  ace  significant  because  the  multiple  day  problem  was 
not  addressed  in  previous  literature. 

5.3  Identity  Networks 

The  objective  for  this  pcxtion  of  the  research  was  to  develop  and  test  an  identity  network  for  use 
as  a  feature  extractor/input  layer  to  a  classifier.  The  data  compression  and  feature  extraction  curabilities 
of  these  networks  perfcHined  as  expected.  When  using  die  identity  network  extracted  features  as  inputs 
to  a  classifier,  die  features  were  acceptable  for  classification  of  data  that  bad  been  gadiered  on  a  single 
day;  10  classes  could  be  classified  to  a  97.5  percent  accuracy.  Increasing  die  class  size  had  some  effect 
on  the  classification  accuracy  using  die  identity  netw<^  extracted  features,  however,  die  accuracy  for 
30  classes  was  96.7  percent  and  for  49  classes  it  was  92.8  percent  These  results  are  very  reqiectable 
given  the  larger  class  sizes.  However,  die  identity  networks  as  feature  extractors  did  not  perform 
well  when  using  data  collected  over  multiple  days.  In  diis  case,  the  maximum  classification  accuracy 
using  multiple  day  data  was  20.0  percent  for  a  10  class  data  set  which  indicates  that  the  generalizidion 
amiabilities  of  the  identity  networks  over  multiple  days  is  very  poor.  As  such,  these  networics  would 
not  be  useful  for  feature  extractors  in  a  realistic  environmenL 
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5.4  Multi|4e  Class  Recognition 


The  multiple  class  recognition  problem  was  addressed  because  many  of  die  results  in  die  literature 
deal  with  small  numbers  of  classes  and  constrained  data  sets.  The  objective  was  to  determine  how  well 
the  networics  performed  when  the  data  sets  were  increased  in  size,  numbers  of  classes,  and  gadiered 
over  multiple  days. 

The  classification  accuracy  using  a  single  hidden  layer  backpropagation  netwrak  widi  raw  image 
data  as  inputs  performed  well  on  single  day  data.  Results  of  classification  for  310  images  consisting  of 
31  classes,  10  face  images  per  class,  obtained  a  92.9  percent  classification  accuracy  on  die  single  day 
data.  Classification  using  two  days  of  data  consisting  of  620  images  of  31  classes,  10  images  per  class 
per  day,  also  performed  well  fOT  this  network  which  olMained  a  classification  accuracy  of  93.3  percent 
The  drawback  of  diis  network,  when  using  Sun  woikstations,  is  the  amount  of  training  time  required. 
As  the  number  of  classes  and  hidden  layer  nodes  increases  die  training  time  increases  to  a  point  of 
being  iiiqiractical  for  real  world  rqiplications  to  recognition.  For  example,  a  network  using  800  images 
of  49  classes  and  a  hidden  layer  of  250  nodes  required  over  30  days  to  train  on  the  Sun  Workstations. 
However,  using  31  classes  and  62  hidden  nodes  the  network  can  be  trainied  iil  approximately  24  hours 
on  Sun  Workstations  and  this  acceptable  for  realistic  plications. 

5.5  Neural  Network  Extracted  Features  versos  KLT 

The  objective  for  this  portion  of  the  research  was  to  conqiare  die  classificaticm  results  of  networks 
using  raw  data  as  inputs,  identity  network  extracted  features  as  inputs,  or  strict  Kathimen-Lobve 
Transform  (KLT)  extracted  features  as  inputs.  Data  for  diese  comparisons  consisted  of  3(X)  images 
representing  30  classes  for  single  day  cpued  data  and  6(X)  images  of  30  classes  for  two  day  captured 
data.  The  testing  in  this  area  showed  that  using  the  KLT  extracted  features  as  irqwts  to  a  classification 
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network  allowed  for  classification  accuracies  of  97%  on  single  day  C2q>tured  data  versus  93.3%  for 
both  the  identity  networic  extracted  features  and  the  raw  data.  When  using  the  two  day  gadiered  data, 
the  KLT  features  provided  dramatic  increases  in  classification  accuracy  over  the  single  day  data;  53% 
for  the  KLT  extracted  features  versus  5.3%  for  the  identity  netwcuk  extracted  features  and  3.3%  ftM* 
the  raw  data  as  inputs.  The  bottom  line  is  diat  the  neural  netwtnks  using  raw  data  or  identity  networir 
extracted  data  as  inputs  to  a  classification  itetwmk  had  accuracies  as  good  as  networks  that  used  strict 
KLT  coefficients  as  inputs. 


5-4 


Appendix  A.  Source  Code 


This  appendix  contains  the  source  code  for  this  research  effort  The  original  backpropagation 
code  and  tnultilaye<  perceptron  code  was  written  by  Dr.  (Capt)  Dennis  Ruck  (29)  and  trKxlified  as 
it  becanw  necessary.  The  modificadons  included  prxting  the  code  on  Sun  workstations,  adding  a 
symmetrical  sigmoid  update  rule,  irr^lementing  an  identity  netwrak,  and  implementing  a  four  layer 
network,  etc.  The  following  code  is  included  in  this  appendix: 

•  Makefile  The  Makefile  was  setup  to  allow  for  variations  in  the  executable  files  by  setting  or 
not  setting  various  flags;  the  flags  are  self  explanatory.  Understanding  the  Makefile  is  key  to 
understanding  the  code  in  this  iqrpendix. 

•  dkmaiii.c  This  is  the  main  roudne  and  contains  many  opdons  depending  on  die  particular  flags 
defined. 

•  backpropx.c  This  is  the  baciqiropagadon  learning  algorithra  This  code  was  modified  to  include 
a  symmetrical  sigmoid  updata  rule  as  well  as  the  standard  sigmoid. 

•  display.c  This  code  was  developed  to  display  images  on  the  Sun  wc^tadon  displays.  It  was 
specifically  designed  to  view  the  identity  network  training  while  the  training  was  in  progress. 

•  dkiox.c  This  code  contains  the  input  and  ou^t  routines  for  reading  weight  files,  data  files,  etc. 

•  P8.C,  P6X.C,  utils.c  This  code  contains  the  utility  routines  necessary  to  gadier  die  data  in  a 
meangfiil  manner.  The  routines  to  conqnite  the  network,  error  statistics,  etc.  are  contained  in 
this  code. 

•  macroe.li,  globab.h,  globals^h  These  are  the  ’Ji’  files,  die  difference  between  globals.h  and 
global8..h  is  that  globals.h  is  included  in  all  files  external  to  die  main  routine,  dkmain.c. 
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•  inakedata.c  This  code  is  used  to  take  the  32x32  pixel,  {>  bit  gray  scale,  ascii  format  image  files 
and  create  the  necessary  data  files  for  use  in  the  main  routine. 


A.1  Makefile 

t  Makefile 

#  Created  by  Dennis  Krepp  June  1992 
HOSTsgrinm 

HOME= / cubS / dkr epp 
NEXT.CFLAGS  =  -DNEXT  -bsd 
SUN_CFLAGS  =  -f68881  -DSON 
SUN4_CFLAGS  =  -bsd 
ATHENA_CFLAGS  =  -ffpa  -DSUN 
DEC_CFLAGS  =  -DDEC 

#  The  following  flags  apply  primarily  to  the  mfn.c  program: 

#  -DMPX  causes  data  to  be  partitioned  according  to  training  fraction  input 

#  -DBACKPROP  causes  the  Back  Propagation  training  rule  to  be  used 

#  (note:  if  training,  this  must  be  defined  unless  you 

#  code  up  another  trainning  rule  ) 

#  -DTRN  causes  the  weights  to  be  saved  and  the  program  to  train  a  network, 
f  -DTRNCOTT  causes  the  net  to  train  the  second  layer  of  a  Cottrell  net 

f  -DNOTRM  use  weight  file  weights  for  testing  the  network. 

#  -DLINEAROUT  causes  a  network  with  linear  outputs  (Cybenko  net) 
i  -DSIGMOID  causes  a  network  with  sigmoidal  output  units 

#  -DSYM_SIGMOID  causes  network  with  symmetrical  sigmoid  outputs 

#  _DINP_SYM  forces  input  layer  to  have  a  syn^sigmoid  output 

#  -DSCALEL.ETA  causes  eta  to  be  scaled  by  the  fan-in  for  each  unit 

#  -DNOSCALE_ETA  causes  a  fixed  eta  to  be  used  for  each  weight 

#  -DMRUNS  use  to  perform  multiple  runs  on  database 

#  -DNODE_OOT  prints  output  node  values  for  sampled  iterations  of  input 
I  -DRANDOM  causes  random  grabbing  of  data  vectors  for  use  in  training 

#  -DIDNET  causes  the  net  to  work  as  an  autoassociator  net 

#  (must  define  S^SIGKOID  with  IDNET) 

#  -DRESULTS  causes  outputs  to  be  printed  to  files  vs  stdout 

#  -DNLP  multilayer  perceptron  flag  for  output  and  setup  files 

#  -DVIEW  view  the  input  and  output  images  (lONET  must  be  defined  also) 

f  -DOJA_ID  makes  an  idnet  function  with  linear  outputs  as  suggested  by  Oja 

#  -DWTS  forces  a  weight  file  to  be  saved  after  each  output  interval 
SIGMOID_CFLAGS  =  -g  -DNODEBUG  -DBACKPROP  -DRANDOM  -DSIGMOID  -DMPX  \ 

-DNOSCAI,E_ETA  -DNOTRN  -I. 

IDNET.CFIAGS  =  -g  -DNODEBUG  -DBACKPROP  -DRANDOM  -DSYM_SIGMOID  -DIDNET  \ 
-DNOSCALE_ETA  -DTRN  -DMPX  -I. 

CYBENKO.CFIiAGS  =  -g  -DNODEBUG  -DBACKPROP  -DRANDCW  -DLINEAROUT  -DMPX  \ 
-DSCALE_ETA  -DTRN  -I. 

COTTRELL_CFLAGS  =  -g  -DNODEBUG  -DBACKPROP  -DRANDOM  -DSYM-SIGMOID  -DMPX  \ 
-DNOSCALE_ETA  -DTRNCOTT  -I . 

BIN  >  $ (HOME) /bin/$ (HOST) 

«BIN> . / 

BACKPROP.OBJ  =  backpropx.c 

MFN_CODE  s  $ (BACKPROP_OBJ)  dkmain.c  ps.c  dkiox.c  utils. c  psx.c  nrutil.c 

CC  »  cc 
SYS_LIB  =  -Im 

mlp:  Makefile  globals_.h  macros. h  macros. h  nrutil.h  malloc.h  $(MFN_CODE) 
$(CC)  $ (SIGMOID.CPLAGS)  -DMLP  -O  mlp  $(MFN_CODE)  $(SYS_LIB) 
echo  'Make  mlp  successful!  1  ' 

idnet:  Makefile  globals_.h  macros. h  macros. h  nrutil.h  malloc.h  $(mfn  CODE) 
$(CC)  $(IDNET_CPLAGS)  -o  idnet  $(MFN_CODE)  ${SYS_LIB) 
echo  'Make  idnet  successful!!  * 

cybenko:  Makefile  globals_.h  macros. h  macros. h  nrutil.h  malloc.h  $(MFN_CODE) 
$(CC)  $ (CYBENKO.CFLAOS)  -O  cybenko  $(MFN_CODE)  \ 
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$  (SYS_I.IB) 

echo  *Hake  cybenko  successful!!  ' 

Cottrell;  Makefile  globals_.h  macros. h  macros. h  nrutil.h  malloc.h  $(MFN.CODE) 
$(CC)  $(COTTRELL_CFUAGS)  -o  Cottrell  $(MFN_CODE)  \ 

$(SYS_LIB) 

echo  ‘Make  cottrell  successful!!  ■ 

stat_norm:  Makefile  stat.norm.c 

$(CC)  -g  -I  -o  stat_norm  stat_norm.c  $(SYS_LIB) 

echo  ’Make  stat_norm  successful!!  * 

NG2ruck:  Makefile  NG2ruck.c 

$(CC)  -g  -I  -o  NG2ruck  NG2ruck.c  $(SYS_IiIB) 

echo  ‘Make  NG2ruck  successful!!  * 

clean:  ;rm  -f  ‘.o  core 


A.2  dkmain.c 

/**«*****««*•*«******•*******•***««*•••*««****«*«*«*»•••***«*•**« 

dkmaitt.c:  Multilayer  Peicepuvn  Program 

Writtea  by:  Dennis  W.  Ruck  (DS-90).  AFTT/ENG 
Modified  By:  Dennis  L.  Krepp  (GE-92D),  AFTT/ENG 
Modifications: 

1.  Modified  to  allow  different  teaming  rales  (eta’s) 
for  each  layer  of  weights. 

2.  Modified  to  run  as  an  Identity  (autoassociator)  net 

3.  Modified  to  work  as  a  Cottr^  net 

4.  Modified  to  use  a  syrrmretrical  sigmoid  if  desired 

5.  Modified  to  print  output  node  values  to  screen  if  desired 

6.  Modified  to  print  the  images  to  the  screen  if  using  the 
-DIDNET-Dymw  options 

7.  Modified  to  work  as  a  4-layer  classification  net,  which  a 
slight  twist  to  the  Cottrell  net  in  titat  tire  classification 

net  is  a  two-layer 
backprop  versus  a  one  layer. 

S.  Modified  the  tmiLpte  runs  option  of  the  code  for  SUNs 

#include  <stdioJi> 

#include  <sigiial.h> 

#iiiclude  <ctype.h> 

#ifndefNEXT 
#include  <inalloc.h> 

^Ise 

extern  char  *  mallocO; 

«eiKUf 

iifdef  RESULTS 
«defiiie  RUN  JTLE  run  Jile 
Itelsc 

#defineRUNJTLEstdout 

^ndif 

#ifdefMLP 

«define  PREHX  ‘mlp‘ 
iendif 

«ifdefIDNET 
Mefine  PREFIX  '  idnet  * 

#define  SIZE  32  /*  this  is  the  image  size  (te.  32x32)  4 

tendif 

#ifdefTRNCOTT 
«deiine  PREFIX  ‘cottrell* 

#endif 

#ifdefTRNKREPP 
«define  PREFIX  ‘dlk‘ 


«endif 


iifdefUNEAROUT 
Mefine  PREFIX  ‘cybenko* 

#ifdefNODEjOUT 

«defiiie  SHOWjOUTPUTS  outiile 

«eii^ 

#iiichide  <inacios.h> 

#iiiclude  <globals-h> 

int  *ivectoi(); 
double  *dvecto«(); 
double  gaussianO; 
char  *gkjtoken(); 
char  •makeJileO; 
void  menuO; 
void  displayJnpuU): 
void  displayjOutputO: 
double  onesjiomializeO; 

static  char  stop jiaine[]  > 'stop_file.dat*; 

FILE  *setup; 

char  datJemp[20]; 

char  wtsJeiiip(20]; 

char  wts^leiiaine[20]; 

main  (aigc,  aigv) 
int  aigc; 
char  *ai8v[]; 

{  .  . 
struct  sigvec  ivec; 
char  c(miiiiand[80]; 


iifdefMRUNS 

#dcfine  SEEDNAME  *  seeds,  dat* 
diar  «tunjoot,  •wttjoot,  •zoojuune; 
int  staiUnumbier,  eixljiumber, 

FILE  •seedJile; 
char  zooxoiniiiaDd(2S6]; 
char  «iunjiaine; 
char  tenip{20]; 

int  fhum; 

#eiidif 

HLE  *ron  Jile; 

char  •wtsjuune,  *dauiaiiie; 
int  tng.vector,  sample; 

double  err,  acc,  tstxrr,  tstncc; 

RLE  astopJiie; 

FILE  *outJile; 

HLE  aplotJile; 

FILE  *idiietJile; 
char  inpJine(256]; 

char  «S^PJ^LE,  ♦NODEJILE,  •PLOTJTLE,  «RESULTSJ1LE; 
int  i.j,k,  target,  m; 

h***  Crate  the  lUe  strings 

SETUPJ7LE-  makeJiie(  'setup',  PREFIX); 

NODEJILE  -  makeJikK  PREHX,  'nodes'); 

HjOTJILE  « makeJile(  PREHX,  'plot'); 

RESULTS -  makeJile(  PREHX,  'results'); 


h****  mi  user  what  type  this  program  is  **** 

This  can  be  added  to  the  output  Bk  later 
if  necessary 

****«**«**«**«*«*«4,**««*«**««««««**«««*«««***««# 


«ifdef  RESULTS 

RUNJlLE-foperKRESULTSJlLE,  'w'); 
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ieiidif 

#ifdefMPX 

fprinti(RlJNJ=ILE,*m£n;  MPX  flag  defined. \n*); 

#endif 

«ifdefIDNET 

^rint^RUNJlLE,*m£n:  IDNET  flag  defined. \n*); 
«ifdefHOOX 

fprintf(RUNJlLE,‘m£n;  HOOX  flag  defined. \n*); 
#ifdefBACKPROP 

fprintf(RUNJ’ILE,*m£n:  BACKPROP  flag  defined.  \n*); 
#endif 

#ifdefLINEAROUT 

fjptintf(RUNJlL^*m£n:  LINEAROUT  flag  defined. \n*); 
ftendif 

#ifdef  SIGMOID 

fprintf(RUNJlLE,'m£n:  SIGMOID  flag  defined. \n*); 

4en^ 

«ifdefSYM.SIGMOID 

fprintf(RUNJ’ILE,*m£n:  SYM.SIGNOID  flag  defined. \n*); 
#endif 

«ifdef  RANDOM 

^^^gtotfi(RUNJ?LE,*m£n:  RAMDOM  flag  defined. \n*); 

fjprintf(RUNJ’ILE,*m£n:  RANDOM  flag  NOT  defined.  \n*); 
ftendif 

«ifdefMRUNS 

fprintf(RUNJlLE,*in£n:  MRUNS  flag  defined. \n‘): 

Hendif 

#ifdefSCALE^A 

fprintfi(RUNJ’ILE,*ffl£n:  SCALE.ETA  flag  defined.  \n*); 
#eiidif 

#ifdefTRNCOTT 

^ntf<RUNJlLE,*m£n:  TRNCOTT  flag  defined. \n*); 
fleadif 

#ifdefTRNKREPP 

fprintf(RUNJILE,*n£n:  TRNKREPP  flag  defined.  \n*); 
iendif 


/*****  Opea  tbe  soup  fife  **•«¥ 

if  ((setup  >  fopen(SEfUPJlLE,  'r'))  —  NULL) 

primf(*I  can't  open  the  input  file'); 
cxit(-l); 

} 


/«•«**  Read  setup  fife  fmneuype  «*•*¥ 
f$canf(setup.  '%d*,&neuype); 
fskipJiiie(setup); 

/*****  Reading  setup  Sle  *m*m4 
fscaiif(setim,  *%d*,&inituLseed); 
fskipJine(Ktup); 
fsca^setup,  *%d*,&paiuseed); 
fsUpJine^etup); 
fsca^setup,  *%d*,  &trn.seed ); 
ftkipJiiie(setup); 


/. 


•«**•  Get  fifeoame  to  save  weights  to  ****4 
fgeMfwtsueinp^.  setup); 
fikip.liiie(setiip); 
i«0: 

while(!isspace(wtsjeinp(i]))  i-t-f; 


wtsJempTilB  '\0'; 
wts  Jiame  ■  wtsJemp 


/«*««*  Reading  setup  Sle  ****4 
fscanf(setup.  *td*,  AmaxJteiatioiis); 
fskipJiiie(setup); 

fscattf(setup,*%d  %d  %d  %d*^umJnputs.Ahidejoiie,ftliideJwo.Aminu>utputs); 
fsknJiiie(Mtup); 

ftrJist « ivectoi((MiuinJnputs-l); 
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mean  -  dvector(OjiiiinJnputs-I); 

$d  «  dvector(0,iniiaJnputs—  1); 
inpuunask  «  dvectoKO^numJnputs—  1 ); 

/*****  Get  Bteaame  to  lead  data  from  ****4 
foets(dalJeiiip,20,  setup); 
twpJiiie(setup); 
i«0: 

wlii]e(!isspaoe(dauenip[i]))  i-t-*-; 
daLtemp[i]B  '\0'; 
dauuune  >  dauemp; 


/**•*«  Read  setup  Bk  ****4 
fscanf(setup.  *%d*,&outputJiitervaI ); 
fsldpJiiie(setup); 

£scanf(setup,  RBALJ’MT,  &etaJn ); 
fskipJiiie(setup); 

ftcaiif(setup,  REALJ’MT,  Aeiajout ); 
ftkipJine(setup); 

fscanf(setup.  REALJ^,  &etaJllJH2 ); 
fskipJiiie(setup); 

fscanf(setup,  REALJ’MT.  Aalpha ); 

fskipJiiie(Mdip); 

fsca^setup,  *ld',  &baichjize ); 

fskipJineCsetup); 

fscanf(setup,  REALJ’MT,  &ini.ftac ); 
fskipJiiie(setup); 
nonnjype*— 1; 

f8canf(setup,  *  %d' ,  &noniUype ); 
fskipJliie(setup); 


«ifdefMRUNS 

fscaiif(setup,  *%d  %d',  Astait-uumber,  dcendJiuiiiber );  fskipJiiie(setup); 
#eadif 


/**  C3ose  the  Setup  6k  *4 
fclose(setup); 

#ifdefIDNET 

lempjoutputs  >  nunuoutputs; 
nuiujoutpuls  a>  numJnpuis; 

#eiKlif 

/*****  INTT  THE  NET  *•♦•♦••••¥ 

iniUKtO; 

I*  fpriatKRUSJlLE,  ’ynUsiae  %d  features  ia  %s.  numJnputs,  daLaame)?/ 
kx>pKnumJopuls)  ftr Ji8l[i]  >  i; 


CREATEJlLBf  stpp J!ie.  stopjiame,  "mfa"); 

fprintff  stopJil^  'Delete  Uiis  Bk  to  foice  a  save  of  tbe  wejgfats.Xo”); 

/close  (stopJik); 


I*****  READ  THE  DATA  *************4 
ieadjdat(  daLname );  h*  function  found  in  <adox.c  *4 
fpiintfCRUNJlLE,  *\n%d  vectors  read',num-vectois);fflush(stdout); 

iifdef  RESULTS 
fclose(RUNJFILE); 
iendif 

iifdefMRUNS 

if  ((seedJile  -  fopen(SEEDNAME.  'r'))  —  NULL) 

printf[*I  can't  open  the  seed  file'): 
cxit(-l); 

} 
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loopK8taitJUiinber)fscaiil(seedJi]e,  *%d  %d  %d',  AinitiaLseed, 

Apart  Atm  jeed ); 

/*•  MuUipk  RUNS  loop 

foKfnum  «  staiUtuinber,  fnum<end  jiumber.  fnum-H-)  { 
iterations  «  0; 

/**  Create  the  file  nmaes  *4 

/•  sprittUdemp,  "%s9d%s%s". PREFIX,  tmun, “RUN"): 
niajuune  « lemp;4 

spiintf(temp,  *%s%d%s%s*. PREFIX,  faum. ' .  *.  ’tn'S*); 
wts  Jiame  >  temp; 

/*  CREATEJFIIm  ninJik,  firajuune,  );4 

£scanf(  seedJile, ' td  %d  %d*.  Ainitial  .seed,  Aparuseed,  Atni-seed ); 
#endif 

Z***************************************** 

butializeAead  the  weights  and  thresholds 
-calls  ulils.c  ->  gaussian 


#ifdefTRN 
srand4^  inhiaLseed ); 

loopKmiin.states)  xhA[i] «  gaussiaii(  0.0, 1.0 ); 
lunpxjtfeiO; 

«endif 

#ifdefNOTRN 

read.wts(wtsjiafiie); 

#eiidif 

iifdefTRNCOTT 
8rand48(  inhiaLseed ); 
readjcottrelLwtsCwtsjiame): 
hendif 

«ifdefTRNKREPP 
snnd48(  inhiaLseed ); 
readJtrepp.wts(wtsjianie); 

«endif 

k)0|rit(neuype) 

loopKLayerfkl-^nuinjOutpuQ) 

loopi(LayeT(kl-»ouinJnputs) 

^  Layer(k]-*wjoldm[i]  -  Layer(k]-*w(jl[i]; 
Layet(ki-»dw(j](i]  •  0.0; 

^  Layet(k]-»theta^ld(i]  3:  Layer(k]-»tlieia(i]; 
Layer[k]-*dtheta(i) «  0.0; 


/• «  Partition  the  data  into  training  and  testing  sets  *4 
iifdefMPX 

parthkHK  tm  Jrac,  parLseed ); 
itendif 

mfdefIDNET 
idneLpaitition  (  patLseed); 
dendif 


tNfndefNOTRN 
/*«*«*  Noanaliiatioo  of  Data 
0:  No  Nonnalixation. 

I:  Gaussian  Nomalizatioo. 
2:  Nonnaliae  to  nage  [-1,1] 
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Select  Nomialualioa  Criurioa  in 
setup  Sk 

twitch  (norauype)  { 
case  0:  biw; 
case  I:  gaussianjionnalize  (); 
break; 

case  2:  gaussianjioniiali;^); 
iiiax.varee  >  ones  JiomializeO; 
break; 

default;  printfC Invalid  selection. \n'); 
ejut(-l); 
break; 

dendif 


#ifdefNOTRN 

/**  Nonnalue  all  data  using peeviously  getteraied  meaa/sd  *d 
IoopKnuni.vecton)  { 
loopjKnumJnputs)  { 
db.li[i]U]  -  (dbJn(i](j]-iiiean(j]V9dUI; 

) 


«fdefIDNET 

/*  now  divide  all  data  by  the  nuu.  value  d 
priotf(*\niiiax_value  s  %f  *,  niax.vahie); 
if  (niax.value  ^  0.0) 


.iKniim.vecton) 
lobpj(numJoputs) 
dbjn[i](j]  s  (dbJn(i][j]liiBX-value); 


dLidif 

iendif 


«ifdef  RESULTS 

RUNJILE»fopco(RESULTS-nLE.  ’a*); 

4en^ 

iifdefIDNET 

loopij(iniin.vecton,  tempjoutputs) 
idjout(i]0]»dbjout[i]|j]; 
loopij(nuiii.vectors.  nuiojoutpiits) 
dbjout[i][j] «  dbJn(i](i]; 

fprintRRUNLFILE,  ‘XnNomalized  IDNET  output  database  initializedXn*); 
dSendif /**  end  IDNET  ifdef*d 


h*  Print  header  for  RUN  Sle  *4 
fprintRRUN  Jn.^* \n*); 

fpriiitf(RUNJ7LE,*Wt  Selection  seed  (initial.seed) :  %u\n\ 

Db  Partition  seed  (part.seed) :  %u\n\ 

Training  Vector  Selection  seed(tm_8eed) ;  %u\n*, 
initial  .seed,  part-seed,  trnjeed ); 
forintRRUN-FlLE,' Weights  file;  %s\n',wts.naine); 
fprintf<RUN.FILE,'Netvirork  Size:  %d-%d-%d-%d\n*. 

nuitt-inputs,  l^jone,  hide-two,  nunuoutputs ); 
forintffRUNJ’lLE,* Source  database:  %s\n*,dauiaiiie ); 
f^tf(RUN.ilLE.*Training  Rate  (eta_in) :  %g\n\ 

Training  Rate  (eta_out) :  %g\n\ 

Training  Rate  (eta_Hl_H2) :  %g\n\ 

Momentum  (alpha) :  %g\n\ 

Batch  Size:  %d\n*,eta-iii,etajout,eia.Hl.il2,  alpha,  batcb-sue); 
forintf(RUN.FIL^*Features  Used:  *); 
fprintf(RUNJlL£,'All  Features  in  Is.Xn'.daUiaine); 

$rintRRim^LE.*\n'); 

’Fraction  vectors  assigned  to  training:  %g\n',tni.firac); 
’Normalization:  %d\n’,nofiiUype ); 


8tais(  AeiT,  ftacc,  TRAIN ); 
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sUtt(  ftlstm,  Am  ax,  TEST ); 

^prinlf[RUNJTLE,*I:  t6d.  ERR:  %6.4e,  ACC:  %6.4e.  TSERR:  %6.4e,  TSACC:  %e.4e\n'.itentioiis. 
err,  aoc,  tstjen;  tsLacc ); 

ploufile  -  fmetKPLOTJTLE,  ‘a*); 

tprintRplotJile.  *%6d  %6.4e  %6.4e  %6.4e  %6.4e  Vn’.heratioiis, 
err.  aoc,  tst-err,  tsUHx ); 
fclos^ploUile); 

«ifdef  RESULTS 
fck)se(RUNJTL£); 

#eiidif 


«ifdefNOTRN 

«ifdefIDNEr 

idnetiile  >  fopenCmlp.idnet  .dat*.  *w*); 

forimRidiiet-filc.  *%d\n%d\n*,Layerlll-»numjwapiits.  teinpjOutpu(s);  loopKiniin.vectors) 


foopkCouinJnpuls)  Layetliieuype-l]-»X[k]  *  dbJn(j][k]; 
compulcjOutpiitO; 
fjpfintRidnetJile,  *%d  j); 

lo(mKLayer[l]-«nufiijoulpuls) 
f^tRidnetiile.  ■%!£  *.LayeT(l]-*Y[i] ); 
looK(teiiu>UNitputs) 
roruitRKlnctJte.  *%lf  ',idjOut(j](k] ); 
fpruMRidnetiile,  '\n*); 


fclose(idiieL^le); 

iencW 

exhC-l); 


h*  minittg  loop 

mitstaie(  tm  jeed.  stale.  STATE^IZE ); 

fifndefNOTRN 
save.wts(  wts  jume.  dauunne ); 
#eiidif 

/«  sweqas  loop  ¥ 

kiopK  (maiuieralioiis/iiiiiuiiiHl )  { 

h*  updme  loop  *4 
k)opj(nuiiUiii)  { 


«ifdefRANIX»« 

target  >  tmJist(niidoni()%iniiiuliii]; 
ielse 


taraet 

ttendir 


>  liiiJisl(iteratioiis%auiiUni]; 


(vec  Jiuiii(iii]  mm  target) 
break; 


cunent-vector  >  tiig.vector  ■  m; 

loopk(iiiiinjnputt)la]«r(neUyiK-I]-»X[k]  -  dbJn(tng.vector](k]; 
loopk(miinjouiputs)  Djout[kI «  dbjoiil[tiig.vecxor](k]; 


!***  ampme  the  outputs  ***4 


compoiejoutputO; 


DimlayinugesifIDNETtkSaed*4 

iifdefIDNET 

if  ((keralioas  %  (iiitXIO.O*outpatJalerval))  0) 

display  JnpuK  tng.vectOf;  inax.value.  SIZE); 
dispb^jOutpuK  max-vahie.  SIZE); 


^•••••••a******  NODE  OUTPUT  ***************4 

«ifdefNODEX>UT 
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SHOWjOUTPUTS  -  fopeii(NODEJILE.  'a* ); 

/****  prims  inputs  Mod/broittpuls  up  to  a 
maximum  of  10  nodes.  This  can  be  changed 
toailnodesbycbangiimtbe  "iTstatemem 
betowforthe  variame  sample' 


if  ((iieiatioiis  %  (intX1.0«oiitpiitJnlervaI))  »0){ 

if  (nunuoutouts  >  10) 
sample  s  lO; 
else 

sample  «  mim-outputs; 

fpriniKSHOWjOUTPUTS.  *  Iteration  %d:  Randcm  vector  td;\n‘,itendoas,tatBet); 

fprimf(SHOWjOUTPUTS, '  Desired  Outputs:  *); 

loopk(sample)  fjprintfCSHOWjOUTPUTS. ■  %4.3f4bjmitImg.vector][k]); 

«ifdefIDNET 

fj^tffSHOWjOUTPUTS, 'Xn  IDNET  Outputs  :  *); 
loopk(sample)  fprintf(SHOWjOUTPUTS,  *  %4 . 3  f  *.  Uyer(0)-»Y[k]); 
itelsc 

foriiitl(SHOWjOUTPUTS,  "Xn  Actual  Outputs  :  *); 
loopk(sample)fprintf(SHOWJOUTPUTS.  ■  %4.3£*.Layerf01-»Y[kl); 
iendif 

fprintffSHOWjOUTPUTS,  *  XnXn'); 

} 

fclosc(SHOWjOUTPUTS); 

#eiidif  /•*••♦*•  ESDNODEJOUT********4 

«ifdefBACKPROP 

backprop(alpha); 

tendif 

hetaiioiis-M-; 

if(iiefalioiis>maxJterBtioas)  bieak; 


«ifdef  RESULTS 

RUNJILE -  fopeiKRESULTSJlLE,  'a'); 

#60^ 

if((ileratioiis  %  outpuLinlerval)  ^  0) 

^  stats(&efT.ftacc. TRAIN); 
stats(  AtsLen;  Aist-aoc,  TEST ); 

fpriiitf(RUNJFILE,'I:  %6d,  ERR:  %6.4e,  ACC:  %6.4e.  TSERR;  %6.4e,  TSACC:  %6.4eXn*.  teratioiis,  ert 
acc,  tsueiT.  isLacc ); 


pkxJile >  fqpenfPLOTJlLE,  *a*); 

^printf<ploLjnle,  *%6d  %6.4e  %6.4e  %6.4e  %6.4e  Xn*. itemioas, 
eiT,  acc,  tstxrr,  tsLacc ); 
fclos^plotJile); 

«ifdef  RESULTS 
fclose(RUNJrILE); 

«eodif 


fifndefNOTRN 

save.wts(  wts  jiame,  dauname ); 
indif 

M  tnke  weights  to  die  after  eariiepo<±  4 
«ifdefWTS 

spf^wtsJileiiame.  *%std*.  wttJuune.(imXiienlioiis/aitpuLJnterval)); 
spriittf(commaiid,  ‘ts  %s  ts*.  *cp‘,  wttJiame,  wts^kaame); 
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sysKnKoomiiiiid); 


) 

«fdef  RESULTS 

fck>ie(RUNJTLE); 

iendif 


}  f**  ead  opiate  loop 
iRiteMk)iis>niixJierations)  break; 
}  /**  eod  sweeps  loop  *<¥ 

«ifiDdefNOTRN 
save.wts(  wts  jiame,  dauname ); 
iendif 


iifdef  RESULTS 

RUNJTLE-fopnKRESULTSJTLE.  ‘a‘); 
iendif 


iifdef  IDNET  h*  Display  last  image  *4 
display  JnpuK  tng.veclor,  niax.mie,  SIZE); 
di^byjotttput(  max.vahie,  SIZE); 


ttats(  tun,  Aaoc,  TRAIN ); 
stait(  tuatjcn,  Atst  anr,  TEST ); 


fprinlf(RUNJTLE,*I:  %6d,  ERR:  %6.4e,  ACC:  %6.4e,  TSERR:  %6.4e,  TSACC:  %e.4e\n*.itenliottS, 
eir.  aoc,  tsuea,  tstacc ); 

plotJle ■>  fapenfPLOTJ’ILE.  *a*); 

fprintKplotJnle.  *%6d  %6.4e  %6.4e  %6.4e  %€.4e  \n*,itefatk»s, 
en,  acc.  tsLeiT.  tstaoc ); 
fclose(plotJile); 

iifdef  RESULTS 

fc)ose(RUNJTLE); 

iendif 


/**  write  bidden  node  output  (0  a  file  for  mlpcrainiafW 
IKfdef  IDNET 

idnet^k«fopen(*inlp_idnet.dat*,  'w'); 

fprintf(idnetiUe, ‘%d\n%d\n',Layei{l]-*nunuoutpuis,teinpjoiitputs);  loopKDo>»-vectan) 

joopkfnumJnputs)  Layer(neuype- 1  ]-»X[k] «  db Jn(j]P;]; 
compuiejoutpiitO; 
fpiiniflidnetiile,  *%d  ',j); 
loqpKLayett  I )— tnunijoutoiiu) 
fpftetRidneLfile,  *%lf  *,IjyettI]-»Y[iI ); 
ioowieiiwjoutputs) 
nriniKidnre^le, '%lf  idjoat(j]Pc] ); 
fpni^kfaietJle.  *\n*); 

/closeCidnetJle); 

iendif 


switch  (nomuype)  { 
case  0:  biea;; 

case  1:  ganssianjonnomialize  0; 


iifdef  RESULTS 
fclose(RUNJTLE); 
iendif 


«ifdefMRUNS 

)  !**  end  maU^  nms  toop  *4 
Mendif 

difdefvmw 
systemCm  *.red*); 
systemCrm  temp.**); 
systemCnn  tenpIN*); 
systemCrm  taqiOUT*); 
systemCrm  *.rle*); 
iendif 

«if(lef  RESULTS 
fclose(RUNJlLE): 

«endif 

} 


A.3  backpmpxx 

/***************m 


badqmpx.c:  Routines  supporting  the  bsdqprop  update  miefv  training 
multiuyw  petcqttrons.  This  versionwoikswitnamaary  numbers  (flayers 
and  uses  the  Layat]  amy. 

Demiis  W.  Ruck,  DS-iUD 
AFTT/ENG 


finchide  <stdk>Ji> 
finclude  <m«hii> 

finchide  <macmJi> 
INnckide  <globabJi> 


void  compute  jdelsO; 
vtrid  compute  .detaQ; 
void  badqpiopO; 
void  badcpfopxO; 

void  computejdebO 

i _ _ 


coamMjdels:  computes  tie  deltas  fora  network.  Uses  neuype 
as  pobal  input  to  determine  what  byers  to  compme  for. 


•4 


switch  (neuype)  { 
case  1: 
case  2; 

case  3:  compute  jdebxQ; 
break; 

default;  Q>rintf(sidetr,  ‘compute.dels:  can't  c(xq>ute  deltas  for  \ 
net_type  «  %d.\n*,  neuype); 
exit(-l); 
break; 


) /**endcomputejdels()*4 

h***m***a****************»»*m»****a****m4 

void  conqxitejdebxO 

double  sum; 
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int  i.j.k; 

loopi(Layer[0]-»iiuiiijOutputs) 

fifdefUNEAROUT 

Layer[0]-»deKi]  *  (Djout(i]-Layer[0]->  Y[i])*LayeifO]-*niask(i]; 
iendif 

«ifdefOJAJD 

Laver(0]-»del[il  ■  (Djout(i]-Layer(0]-»  Y(i])«Layei(0]-*inask(i]; 
iendif 

«ifdefSYM^IGMOID 

LayerIO]-»delfi]  -  (0.S)*(1.0  -  (U^0]->Y[i]*Uytf{0]-»YIi]))* 
(Djoutui— LayeifO]-*Y[i])«uy(^0]-*iitssk(i]; 

iendif 

«ifdef  SIGMOID 

UyeitO]-»delIi] «  UyeifO]— Ym«(1.0-Layef(0]-»Y[i])  * 
(Djouai]-Uyer(0]-»Y(i]>*Layer(0]— masklil: 

iendif 

foiO^  1  dc<  netiypc'Jc-H-) 
loopj(Layeifk]-»nuinjoutpiits) 
sum =0.0; 

loopKLaycffk—  1  ]-»nuiiLjOUtputs) 
sum  +=  Layeifk— 1]— *del[i]*Layet(k— 1]— *wfi][i]; 
iifdefSYMjiGMOID 

Layer(kl-*dellj] « (0J)*(1.0  -  (LayerOc]-Y[j]*Uyer(kl-»Yljl))  • 
tum*Layef(k]-*iiiaskrj]; 

iendif 

iifdefOJAJD 

Uycrlkl-dcim  » (0.5M1.0  -  (Laycilk]-*Ya]*Uyerfkl-»Yai))  * 
sum«Layef(k]-*maskfi]; 

iendif 

iifdef  SIGMOID 

Layettkl-^delUl  ■  L^iOtl-*Yljl*(l  .0-Layeitkl-»Ym)  ♦ 
sum«Layer[k]-*iiia^]; 

#6fidif 

fifdefUNEAROUT 

Layet(kl-»delljl  -  Uyei(kJ— Y(jl*(1.0-La)ierlkl— YO))  ♦ 
8um*Layettk]-»iiia^]; 

iendif 

} 

} 

}  /«*  end  computejdelsxO 

void  backpropfalpha) 
double  alpha; 

z************************************************************* 

bad^prop;  Implement  the  backpmp  weig/u  update  mk. 

INPUT:  aeuype 


switdKnetiypeX 
case  I: 
case  2: 

case  3:  backpropx(alpha); 
break; 

default  :  fprirnKstderr/backprop:  Illegal  net.type  «  %d\n*,neuype); 
exiK-l); 

} 


void  backpropxfalpha) 
double  alpha; 
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badkpmpx:  Backpn^  update  for  a  multilayer  net. 

INPUTS:  Layerfl  Djout 
OUTPUTS:  Uyera 

***«*«****«««********«*********«**«**«**«*««******«*««*****«*4 

double  tW,  Olieu; 
double  leu,  lalpha; 
lot  I,  j,  k.  epoch; 

lalpha  >  alpha; 

batchxm-t-f; 

computejdels(); 

/**  Update  all  layers  *4 


#ifdefTRNCOrr 
loopk((neuype  - 1)) 
denidif 

iifdefTRNKREPP 
loopk((neuype  -  I)) 
iendif 
«ifdefTRN 
loopkCueuype) 
dcndif 
{ 

difdefSCALEXTA 

leu  «  Layef(k]-*eu/kloubleXLayei(hl-*numJinpuls); 
ieodif 

«ifdefNOSCALE^A 
leu  ■  Layef(k]-»eu; 

«eiidif 

#ifdefIDNET 

epoch  B  (hentioaa/iuiiLtni); 
in(epoch  <  ^  (k  BB  0)) 
leUBOJ^oatXI  ■•■epo^); 
if((epoch  >  2S)  (k  B.  0)) 

kUB  (leu/10.0); 

h  if((iteratiotts  %  auautm)  **  0) 

printl('\a  epoch  *9U,eta*  %/\a*,epodMeu); 

#eodif 


/**  Ompute  the  cbaitge  in  the  weights  *4 
loopj(Laye((k]-»nunu>utputs) 

if(La;«(0c]->niaskU}»OFF)  continue; 
loopKLiva(k]-»nuniJnpuU) 

^Layet(k]-*dw[ilQ]  +«  Layei{k]-»del(i]*Layei(k]-*X[i]; 


Lq«i(k]->i 


} 


dtheu(j]  -fB  Layer(k]-*delU]; 


/**  Update  the  weights  if  indicated  *4 
IKbaichjcniBBbalchjize) 

loopj(LayeT(k]-»nuinjootputt) 

lf(Layetfk]-»niatk(j]BBOFF)  continue; 
loopi(Lqfer(k]->nuiiiJnputs) 
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tW  •  Lawr[k]-*w[r 
LaywftJ-»  ■ 


[k)-»w[i](j]  +« leta«Lavei(k]-»dwfi](i]  + 

ialpto*(Lay^]-*w[i][j]-Lay^]-*w^k)(i](j]); 
|-«wjoldu][j]  ■  tW; 


Layerfk] 
LayerIk]-*dw[i]G]>0.0; 


frheta  s  Layer(k]-*theta(j]; 

Laye»(k]-*tneta|j]  lcia*La;wt{k]-*dttieU(j]  •»' 

lalfna*(Layer{k]-»theta(j]— Lay^]-*tlieta.okil)]); 
Layeifk]-»tlietaloldn]  >  tTheta; 

Uyer{k]-»dthett(j]-0.0; 

} 

}  h*  end  we^ht  update  *4 


}  !**  end  layer  update  *4 
if(batchjcnt»>iMtclLjize)  batducnt  ■  0; 
}  /**  end  backpropxO  *4 


A.4  dkioxx 

/*********0m************************************** 

dkiox.c:  Utility  Functkm  to  support  perceptnm  I/O 

Writteaby:  Dennis  W.  Ruck,  DS-90D  AFTT/ENG 
Modified  by:  Dennis  L  IGej^,  GE-92D  AFTT/ENG 
Modifications: 

1.  readjdat  modified  to  save  results  to  files 

2.  leatLwts  modified  fa- a  CotaeU  net 

«*««*«**««**«*«***««**#*4i**««***«**«**«**««**«**4 

IKnctude  <8tdioJi> 

#iiiclude  <s(rings.h> 

#iiiclude  <nutta.h> 

#iiKlude  ‘macros. h* 

#include  ‘globals.h* 

#ifdef  RESULTS 
«defiiie  RUN  JFILE  stdout 
#else 

«define  RUN  JTLE  stdout 
ttendif 

double  draiKMSO; 
double  gaussisnO; 
char  •  geuokenO; 
double  «*dinatrix(); 
int  *ivectofO; 

/*«  Functkms  Exported  *4 
void  leaiLdatO; 
void  tead-wtsO; 
void  leadxotuelLwtiO; 
void  save.wts(); 
void  save.dat(); 
void  makejnobO; 
void  tavejifniO; 


/,«•«•*  VOIDSAVEWTS 
Mve-utf  uses  the  foliowing  global  input  variables: 

mmUnputs,  bidexne,  bideawo,  numjoutptds 

iterations,  niax.value 

iaitiaLseed  (used  in  setting  wdgbts) 

paitjeed,  trtLseed 

muiLstates 
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mjype 

wtsjype 

flrJist 

ttorttUype 

tmJnc 

domiaattUseasor 
ttunLdir,  dirJistfJ 
nuaung,  ngJistf] 

t^<t^*m*m***m************if*****************************************4 
void  save.wts(  wts  jiame,  dat-name ) 
char  owtsjiaiiie,  *daUiaine[]: 


HLE  *wts; 
char  oteinp,  ^tkn; 
int  i>0; 


/•  ixmti("lq>iox.c-  save.wts  \tt’)^ 


CREATEJ1LE(  wts,  wtsjuune,  *  save.wts* ); 

fprintf(  wts,  ■VJTS.TYPE:  %d\n*,  wtsjtype ); 

glf(wts,  '%d  —  Number  inputs\n*,nuinJnputs ); 
tf(wts,  ‘%d  —  Number  HI  nodes\n*,liidejoiie): 
tf(wts,  ‘%d  —  Number  H2  nodes\n',hideJwo); 
tf(wts,  '%d  —  Number  outputs\n',numjout|)uts); 
fprintfif  wts,  'td  —  Number  iterations\n*, iterations ); 
fprintf(wts,  '%d  —  Initial  wts  seed\n*,iiiitiaLseed): 

I***  Now  save  the  weights  **4 

h**  Use  aa  type  **4 

/««« 

Type 

0  :  Single  peroeptron 
I  ;  One  layer 
2  :  Two  layer 
3  :  Three  layer 

**4 

hm*  Copy  weights  to  the  state  array  **4 
switch(oeuype)  { 

caseO;  fptintfi(stdenr,' save.wts:  Illegal  net.type  =  %d!\n*,DeUype); 
exit(-l); 
break; 
case  1 : 
case2: 

case  3 ;  mpxJuiferO; 
break; 

default:  fptintfl(stderr,* save.wts;  Illegal  net.type  =  %d!\n*,iieUype); 

cxit(-l); 

^  break; 

loopi(nunLstates)  fprintf(  wts,  '%e\n*, xbat[i] ); 

/**  Save  additiona/  wts  info  for  newer  wts  files  *¥ 
if(  wts  Jype  —  WTS-TYPE-l ) 

{ 

foriatf(  wts,  *%d  --  Partition  8eed\n*,paitaeed); 
motfCwts,  'td  --  Training  seed\n*,timeed); 
rofintf(wts,  *ts  —  Source  database\n*,daUiaiiie ); 

^rilltfC  wts, 'FEATURES:  '); 

loopKmimJnputs)  fjprintft  wts,  *td\n*,  ftrJist[i] ); 

fenntffwts,  *td  --  Normalization  methodXn'.nomuype); 

(printKwts,  *%f  —  Normalization  ma\_value\n*,(float)inax.value); 
forinlff  wts,  *te  --  Fraction  assigned  to  training\n*,tnJitac); 
uopKnumJnputs) 

Ipriiitft  wts,  *%f  —  mean  ltd] \n',  (float)  inean[i],i); 
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fprimf(  wts,  ■%£  --  sd(%d]\n',(float)sd[i],i); 

} 

} 

fclose  (wts); 

}  /«  end  save.wts  4 


|^^^^*1^^^^m******^^***^*^n^*^^*******^n^****»***m^^*****4 
void  read.wt$(  wts  Jiame ) 
char  *wtsjianie; 


char  *tenip,  *tkn,  first; 
int  i  =  0,  tknJen; 


double  junk; 


/*printf("dkiox.c- teaiLwts  \n");4 


/***  Open  ibe  wts  Sk  **4 
OPENJILE(  wts,  wts  Jiame,  •read_wts‘ ); 

/**  Check  if  the  wtsjype  indicator  is  present  *4 
if(  (first=fgetc(wts))  ^  'W' )  { 

/**  No  wtsJype  indicator  present  *4 

/**  Aiempt  to  push  it  back  onto  the  input  stream  *4 

if(  ungetc(first,  wts)  =  EOF )  { 

fprintf(stderr,  •read_wts:  can't  return  character  to  input  stream. \n'); 
exit(-l); 

wtsjype  =  WTS-TYPEJO; 

} 

else  { 

/«*  wtsjtype  indicator  present  *4 

tkn  -  get  joken(wts);  /«•  getting  remainder  of  WCS-TYPE:  iabel  *4 

fscanu  wts,  'kd', &wtsjype ); 

fskipJine(wts); 

} 

/««*  Read  the  data  **4 

fscanf(  wts,  '%d*,  &numJnputs ); 

fskipJine(wts); 

fscanf(  wts,  '%d*,  &hidejone ); 
fskipJine(wts); 

fscanf(  wts,  *%d',  &hideJwo ); 
fskipJiiie(wts); 

fscanf(  wts,  *%d*,  &nuiiuxitputs ); 
fskipJine(wts); 

fscanf(  wts,  '%d',  Alterations  ); 
fskipJine(wts); 

fscanf(  wts,  *%d*,  AinitiaLseed ); 
fskipJine(wts); 

printf('nuffl_inputs  =  %d  \nhide_or.e  =  %d\nhide_two  =  %d\n\ 
num_outputs  =  %d\niterations  =  %d\ninitiai_seed  =  %d\n\ 
wts.type  =  %d\n',numJnputs,hidejine,ludeJwo,numjoutputs,  iterations, 
initial-seed,  wtsjype ); 

/«««  Now  get  the  weights  **4 
/*«*  First  determine  net  type  **4 

Type 

0  :  Singk  percepiron 

1  :  One  layer 

2  :  Two  layer 

3  :  Three  layer 

**4 

if  ( (hide  JMie  »  0)  AA  (hide.two  0)  AA  (nunijoutputs  »  1 ))  neUype  °  0; 

else  { 

if  ( (hide  j)ne  Ks  0)  AA  (hide  Jwo  n  0) )  neuype  >  1 ; 
else  { 

if  ( (hide-two  0) )  neUype  «  2; 
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else  netJype  «  3; 

}} 

printf(*net_type  =  %d\nMieuyi)e); 
switch  (neuype) 

{  „ 

caseO  : 

easel  outpuUayers&Ll;  break; 

case  2  :  outpuUayer  =  &L2; 

numjlates  =  numJnputsohidejOiie  +  hidejone  * 
hidejoiie^niiinjoutputs-t-  ounuMiQHits; 

break; 

case  3  :  outpuUayer «  &L3; 

nunustates  =  numJnputs*hk)ejoiie  hklejoiie 
hidejoae«hkleJwo  hide Jwo -«■ 
hideJwoanunijoutputs-)-  nunuHitputs; 

break; 


/•*  Read  in  weights  using  the  neUype  *4 

/•«*  Fifst  read  into  xbat  array  **4 
loopi(nunustates)  fscanf(wts.  REALJ^MT,  &xhat(i]); 

/«**  Copy  weights  to  the  layer  structures  **4 
switch(iieUype)  { 

case  0 :  break; 
case  1 :  break; 
case  2 :  lunpl  Jtfer(); 

break; 

case  3  :  kjiip3 JtferO; 

break; 

default:  printf('read_wts:  Illegal  net_type  =  %d!\n*,iietJype); 
exit(-l); 
break; 

} 

h*  Read  in  additkmal  wts  info  fornewer  wts  Sks  *4 
if(  wts^  —  WTS.TYPE.1 )  { 
fscan^wts, '  %d' ,  &patt.seed );  fskipJineCwts); 
fscanft  wts.  * %d*,  Atm jeed );  fskipJineCwts); 
fskipJineCwts); 

temp  s  geUokraCwts);  /««  Slapping  over  ’FEATURES:’  4 
loopiCnumJnputs)  fscanfC  wts,  * %d',  &firJist(i] ); 
fscanfC  wts, ' %d',  AnomLlype );  fskipJineCwts); 

/*  fscanR  wts,  REALJ’MT,  Junax.vahre  );  fskipJine(wts)p4 
fscanfC  wts,  REALJMT,  Ajunk );  fskipJineCwts); 
loopKnumJnputs) 

/scanfC  wts,  REALJ=MT,  Aiiiean[i]); 
fskipJineCwts); 

fscanfC  wts,  REALJMT,  Asd(i]); 
fskipJineCwts); 

} 


} 

fclose  Cwts); 

}  /*  endread.wts4 

hmmmm****  VOID  READ JCREPP.WTS  *************»****»4 
void  readJtrepp.wtsC  wts  Jiame ) 
char  *wtsjiaine; 

^LE  *wts; 

char  *ieinp.  *tkn,  first; 

int  i*0,tkn  Jen,  junk; 

int  j; 

int  1DX>0; 


Opa  the  wts  Sh  **4 

OPENJILEC  wts,  wtsjuune,  *read_wts* ); 
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h*  Cbeckiftbe  wtsjype  indicator  is  picseat*^^ 
if(  (fiisNfgetc(wts))  ft  'W')  { 

/•*  No  wisjype  inaicaior  present  *4 

/**  Atempt  to  push  it  back  onto  the  input  stream  *4 

ifi(  ungeu^fint.  wts)  EOF )  { 

f^printf(stden'.  ■read_wts:  can't  return  character  to  input  stream. \n"); 
exU(-l); 

wts  Jype  =  WTS.TYPEjO; 

} 

else  { 

/«*  wts  Jype  indicator  present  *4 

tkn  =  8euoken(wts);  /«*  getting  remainder  ofWTSJTYPE:  iabel  *4 

fscann  wts.  *%d*. &wts.type ); 

fskipJiiie(wts); 

} 

/•**  Read  the  data  **4 

fscanfC  wts,  'td*.  &nuniJnputs ); 

fskipJjne(wts); 

fscanf(  wts, ' %d*,  &liideu>iie ); 

fskipJine(wts); 

fscanf(  wts, ' %d*,  &juiik ); 

fskipJine(wts); 

fscanfC  wts, ' %d*.  &junk ); 

fsldpJineCwts); 

fsct^  wts,  * %d*,  Ajunk ); 

fskipJbnefwts); 

fscanfC  wts,  *%d*,  &junk ); 

fskjpJine(wts); 


Now  get  the  weights  **4 
/***  Net  type  *  Three  layer  Krepp  **4 

outpuUayer  s  &L3; 

nuiiLstates  «  numJnputs*hidejone  +  hidejone  + 

liidejone*hidejwo  ‘f  hide Jwo -«■ 
hide  Jwo«nuinjoiitputs+  nonuHitputs; 


/**  Read  in  weights  using  the  neuype  *4 

!***  Fast  read  input  layer  wts  into  xbat  array  **4 
loopi((nuiiiJnputs«hidejone  hidejone)) 
fscaitffwts,  REALJ^T,  &xhat(i]); 

/*••  close  the  weight  Sle  **4 
fclose  (wts); 


/***  Copy  weights  to  the  layer  stmctures.  **** 
*•  this  code  was  copied  from  kjnpSjtfer  and*** 
**  was  specialized  to  read  in  the  weights  *** 

**  fora  Krepp  net  setup  ****************4 


/***  Get  first  layer  **4 
loopKhide.i>ne) 

^loopj(nuiiiJnpiits)  Ll.w(j][i]  >  xhat[IDX'4-«-]; 
^LI  .tn^i]  s  xhat(IDX-H-]; 

/***  Get  second  layer  **4 
loopKhideJwo) 

^loopKhidejone)  L2.w(j][i] «  gaussian(  0.0, 1.O ); 
L2.tnka(i]  ■  giussiaiH  ().0, 1.0 ); 


} 

/***  Get  third  layer  **4 
kwpKnunijOutputs) 
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Ioopi(hide Jwo)  LS.wMKi]  -  gaiissuui(  0.0, 1 .0 ); 

L3.to^i]  ■  gaussiaii(  0.0, 1.0 ); 

} 

printf(*\nKrepp  output  weights  initialized  \n*);fRush(stdout); 

}  /*  end  ieaiLkTepp.wts  V 

/^rn*******  VOID  READXXyTTRELL-WTS 
void  iea(Lxottrcll.wts(  wts  Jiaine ) 
char  *wtsjianie; 

^LE  4>wts; 

char  *temp,  *tkn,  first; 

int  isO,tkii  Jen,  junk; 

int  j: 

int  IdX  b  0; 


/***  Open  the  wts  Ok  **4 
OPENJILEC  wts,  wts^name,  *read_wt8* ); 

h*  Check  if  the  wtsjype  indicatoT  is  present  *4 
if(  (firsti:fgetc(wts))  ^  'W' )  { 

/«*  No  wtsJype  indicator  present  *4 

/*•  Atempt  to  push  it  back  onto  the  input  stream  *4 

if(  ungetc(first,  wts)  —  EOF )  { 

^printf(stdeiT,  'read.wts:  can't  return  character  to  input  stream. \n'); 
exit  (-1); 

wtsjype  *  WTS.TYPEjO; 

} 

else  { 

/•V  wtsjype  indicator  present  *4 

tkn  =  jKUoken(wts);  !**  getting  lemainder  ofWTS.TYPE:  label  *4 

fsanK  wts,  * %d*,  AwtsJype ); 

fskipJine(wts); 


/***  Read  the  data  **4 

fscanf(  wts,  * %d*,  &nuinJnputs ); 

fskipJine(wts); 

fscanf(  wts,  *%d*,  &hidej)ne ); 
fskipJine(wts); 

fscanf(  wts,  '%d*,  AhideJwo ); 

fskipJine(wts); 

fscanf(  wt^  ' %d*,  &junk ); 

fskipJine(wts); 

fscanf(  wts, ' %d*,  &Junk ); 

fskipJuie(wts); 

fscanfi  wts, '  %d*.  A, initial -seed ); 
fskipJine(wts); 


/•*«  Now  get  the  weights  **4 

Net  type  ■  Two  uyer  CotueU  **4 
neuype«  j; 

outpuUayer  B  AL2; 

nunustates  *  nuiiiJnputs*hidejone  hidejone 

hidejone*nuinjQutputs-f  nunuoutputs; 

/**  Read  in  weights  using  the  neuype  *4 

/***  First  read  into  xhat  array  **4 
loopKnununates)  fscanf(wtt,  REAl-FMT,  &Jihat[i]); 

/**«  close  the  weight  Ok  **4 
fckwe  (wts); 


/***  Copy  weights  to  the  layer  stnictuies.  •*«* 
*•  this  code  was  copied  Oom  kji^2jifer  and  *** 
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**  was  specialized  to  read  in  tbe  weights  *** 

**  for  a  Cottrell  act  setup  **4>***««*«***«**4 


/••*  Get  Srst  layer  *•¥ 
loopi(hidejoiie) 


{ 

} 


loopUnumJnputs)  Ll.w[j][i]  >  xhat[IDX-H-]; 
Ll.tnka[i] «  xhat[IDX-M-]; 


/***  Get  secood  lam**<i( 
loopKnunuNitputs) 

kx>pj(liidejoiie)  L2.w[i][i] «  gaussiao(  0.0, 1.0 ); 

L2.tneta(i]  s  gaussianT  0.0, 1.0 ); 

} 

printf(*\nCottrell  output  weights  initialized  \n*);fflush(stdout); 
}  /*  eadreadjcottrelLwts^l 


/*4>****  VOID  READ J)AT  *««4i««***»«***«*««««***«***«*«*«**»«*«»*« 

Read  a  database  from  file  dsLJtame. 
mPUTS: 

numJoputs:  determines  how  many  features  to  use. 
firJistlj:  deteimines  which  features  to  use. 
numjoutputs:  used  if  IDNETis  defined. 

OUTPUTS: 
flum.  vectors 
dbJOuHJl] 

dbJnim 

nUBLjOUtpUtS 

vec.jiumf] 

vecjentiyl] 

******m*********t**********»*******m****m*m****t********m***»**04 

void  leadjdatC  dauume ) 
char  *dat -name; 

^ILE  •datHle; 

double  temp; 

int  cat  -  6; 

im  db  inputs,  dbjoutputs; 

int  y,  inpxnt,  went,  esuvectois; 

int  scan.val; 

char  «wcteinp; 


/«*  Open  the  tile  *4 

OPEN J1LE( daufile, dat-nanie,  *read_dat* ); 
iifndef  RESULTS 

fprintf{RUNJlLE, '\nReading  %s:  *,  datJiame);  fHush(ttdout); 
iendif 

/**  Read  number  of  inputs  and  outputs  *4 

fso^  daLfile, ' %d*,  Adbinputs);  fiddpJine(daL.file); 

ftcanK  daufile,  ’%d*,  &dbjoittputs );  fskipJin^daLiile); 

h*  esthnate  tbe  auiaber  of  training  vectors 
find  Ore  number  of  words  and  divide  by  (inputs-Kwtputs-f  IJ  *4 
went-O: 
whiled ){ 

wctoiv  «  geUoken(datJile); 
iKwetonp/NULL)  { 
wcm++; 
frBe(wcieiiip); 
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else  break; 

esuvectors  *  went  /(dbJnputs-Hlbjoutputs-t-1 ); 

#ifdef  DEBUG 

fprintf(siden','Estiinated  number  of  vectors  in  %s  is  kdXn’.datJUfne, 
esuvectors); 

#endif 

/•*  Reset  fiJe  forteadiag  Ibe  vectors  *¥ 
rewinrKdaUile);  fskipJine(datJile);  fskipJiiie(datJile); 

/*•  Allocate  dbJn  and  dbjout  using  the  est.vectois+2  *4 
dbJn  s  dniatrix(0^t.vectors-  l,04iuinJnputs- 1 ); 
dbjout  >  dimarixfOxst-vectors— 1  Anuiojou^ts—  1 ); 
vecjium  s  iyectot(0,est.vectors— 1); 
vec  jentiy  « ivecU)t(().est.vectof8—  1 ); 

#ifdeflDNET 

idjout  s  diiiatiu(0,est.  vectors—  1  .O^joutputs—  I ); 
llendif 

/**  Read  the  tniningAest  vectors  *4 

whU^  1 ) 

/**  get  next  data  pair  *4 

if(  (scan.val=fs(^(  datJile,  * %d*,  &vecjiuin(cnt] )) »»  EOF) 

{ 

num.vectors  *  ent; 

br^; 

Mpxnt  K  0; 

loopi(dbJnpuu) 

^if  ( f5canf(datiile,  REALJ^,  Atemp) »  EOF) 

{ 

printf(*read_dat:  incoDplete  file  -  lacks  full  input  vector. \n*); 
exit(-l); 

u(  (inpxnt<nuiiunputs)  &&  (i »  ftr-^inpjcnt]) ) 

dbJn(cntl[inpjcnt-M-] « temp; 

2x>pKdbjoutpua) 

if  ( fscanfCruLfile,  REALJ’MT,  &dbjout(cnt][i]) »  EOF) 

{ 

printfCreacLdat:  incomplete  file  -  lacks  full  output  vector. \n’); 
eHt(-I); 

} 

cnt++; 

«ifiidef  RESULTS 

ifi(cnt%10«»0)  {^tf(RUNJlLE,‘*');fRush(stdout);} 

#endif 

«ifdef  DEBUG 

piinli(*8canf  s  %d,  ent  =  %d\n*.scan.val,cnt); 

#endif 

}  t**  end  while  *4 


if{esLvecton;£nuiiL.vectofs)  { 

fpfintfi(slderr.*rea<l.(lat:  error  in  reading  file:  est.vectors  =  %d,  \ 
nuauvectors  =  %d\n',est.vecton,auin.vectois); 
exi^-l); 

} 

fcloMfdaufik); 

}^eadiea(Lilat¥ 
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void  uvejdaK  datjuane ) 
char  *dauuune; 

saveulat:  Routine  to  save  a*  Jiat  fUe.  The  puipose  is  to  save  the 
raw  MSF  dbJn[][]  and  dbjOutlJO  arrays  on  disk  because  the  reacLmsfdat 
is  a  very  shw  process. 

INPUTS: 

dbJnnn 

dbjoutlll] 

vecjiuml] 

mmUnputs,  numjoutputs,  aum.  vectors 
OUTPUTS: 

None. 


*«**«*««*««*«*******«**«*««*»*«*«*««««*««*«««**«*««*«*«*««*********^ 


int 

FILE 


printfCkpiox.c--  save_dat  \n*); 


CREATEJ1LE(  dat,  dauuune.  *save_dat‘ ); 

fpfiatf(dat.  '%d  --  nuiiL.inputs\n%d  —  nuiiL.outputs\n*,nuinJopiits, 
numjoutputs ); 

k>opk(nuin.vectors)  { 
fpni^  dat,  *td  *,  vecjium[k]); 
kx>pj(nuniJnputs)fprmtf(dat. '%e  *,dbJn[k]n]); 
loopKnunijoutputs)  fprintfl  d^  ' %e  *,  dbjoutj[K](j] ); 
fjpnntfi(  dat.  *\n* ); 

} 

fck>se(dat); 

}  /•  eodsavejdatd 

void  make  jnesIK  input,  output) 
double  •input,  *out^; 

{ 

-  Procedure  to  generate  test  data  for  training  and  testing 
-classiScationalgotitbtas. 

NOTE:  It  is  assumed  that  the  random  number  generators  have  been 
set  before  calling  this  routine. 

Usesrand4S(seedval)  to  initialine  random  nurttber  generator. 

Output;  Viro  Arrays  of  double.  The  Srst  contains  the  input,  mtd 
the  second  contains  the  desired  outpM. 


idefine  1jOWjOUTO.IO 
«define  HIGH  JOUT  0.90 

int  class; 

double  ladins; 
int  i; 


printK'Icpiox.c—  Bahejaesh  \n*); 


-dnnd48()*6.0- 3.0; 
-dnnd48()«6.0-3.0; 
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CREATEJILEC  xftn.  xfmjume,  'save_xfm' ); 
fpriiitf(xfin,  ’ISd  —  nuQ_inpuCs\n%3d  --  num_outputs\n*, 
Layerlnetiype- 1  ]-*numJiiputs, 
Lqiei(iiet.qrpe-I]-*iiumjOi«puts ); 

/*«  Ptiat  the  aoan$tizaUoa  data 

loopKmunJnpuU)  fpriiiif(xfni,  *%E  tE\n*, iiiean[i],  sd(i] ); 

h*  PriM  weights  date  *4 
loopi(liq«ffneuype-l]-»niiiiijout|iuts)  { 
loopKniniJnpiitt)fjpriiitf(xfni,*%E  ',Lq(tftiiet.^-l]-»w(ilIi]); 
fpni^xfm.  *  %E\ n  \  l^y^neuype- 1  ]-»dKta(i] ); 


fcloae(xAn); 

}  /•  ea(luvejt6n4 
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A.5  ps.c 

/************************»m***m**m***********m**** 

ps.c:  Utility  Functions  to  support  penxptroas 

Dautis  W.  Ruck,  DS-90D 
AFTT/ESG 

***«««***********«***«*«********««**«***»****«««4 

iinchide  <sidioJi> 
iinclude  <iiiathJi> 

#iiicliide  <inacnM.h> 

#iiiclude  <gk)ba]s±> 


«ifdef  RESULTS 
#define  RUN  J’tLE  sidout 
4clsc 

Mefine  RUNJ’ILE  stdout 
Deiidif 

int  findjnaxO; 
void  soitO; 
int  *ivectoi(); 
double  *dvectoi(); 
kmglabsO; 

void  computejOutpuLAomJMlO; 
void  cofiip^j0utputin>fnJ12(); 
void  gaussuuuiioniializeO; 
void  gausaian  junnormalizeQ; 
double  onesjionnalueO; 
double  sigmokK); 
double  t^rtninetnc  jigmoidO; 
void  paimrauO; 
void  stalsO; 


h**********************»**»******4 

int  lLjnp2  jtfeK) 

i,.. 

intu; 

intnE)X>0; 


/*piiBiI("ps.c-  k-rt^iljtfer  \n 

/***  Get  tint  layer  **4 
loopKhidejone) 


kxmKnumJnpMt)  LI.w(j](i] «  xha^IDX■H']; 
LI.theta(i]-xliat(IDX-M-]; 


} 


/***  Get  second  bm**4 
loopKnunuxitputt) 


lejone)  L2.w|j](i1  -  xlut(IDX-H-]; 
[il-xhatPDXW); 


)  h  eodkjap2.xfer4 

/m*********************mm*m*m4 

int  lLJiip3  JtfeiO 


i«a>X-0; 

h**  Get  fyu  layer  *n4 
loopKhidejoae) 
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k>o|^nuinJiiputs)  Ll.w()](i] «  xlut[IDX-H-]; 
^Ll.ueta(i]  ■  xhalilDX'M’i: 


/***  Ga  second  lMyer**4 
loopKhideJwo) 


xhatODX-t-f]; 


/***  Get  tbtd  layer  **d 
k)0|»(nunu>ut|)uts) 


xhatODX-H-]; 


}  /*  eadkJopSjiferH 


/***«4>*«*««*«***«**««******4i«**«*«*^ 

void  stats(eiT,  aoc.  set) 
double  *err. 
double  *acc; 
int  set; 


double  sum  >  0.0; 
int  eiTxnt  >  0; 


int  vlist,  lisuise,  vec; 
int  ij,  tatget; 


if(  set  — TRAIN ){ 
list  s  trn  Jist; 
listjue  a  ounum; 

else  { 

list  a  tsUist; 
lisuize  a  nunList; 

} 

kiopKlisusize) 

^taiset  a  list(i]; 
lo(^nuni.vecton) 
if  (vecjuimU]  —  taiget) 
break; 
vec«  j; 

loopj(nuinJnpuls)  LlJC(j]  a  db  jn(vec](j]; 
compotejOutiNitO; 

if  ( fuxLmaxI  A<tt>jOut(vec][0],  nunuoutnuts) 

^  findj^(  out|iutJayer-*Y,  num.outputs ) ) 

eiTxat-H-; 

loonKnunLoutnuts) 

sum  4a  (dbjout{vec](j]  —  outputJayer-*  Y(j])  • 

(dbjout(vec][j]  —  outpuUayer-*Y(j]); 

} 

ifOisLaize  —  0){ 

*accal.O; 

*eiraO.O; 

return; 

lacc  a  ( (double)  (liscsize  -  etijcnt))/!  (double)  lisLsire); 

*err  a  (OJ  •  sumVC  (double)  li^ize); 

}A*endstats¥ 

void  coiiniulejoutput.ftomjll() 
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{ 

double  sum; 
intij; 


printfCps.c--  coinpute_output_from_Hl  \n*); 


switch  (neuype)  { 
case  2: 

break; 
case  3: 
break; 

default:  fjptiiitf(stderr, 

•compute_output_£ronL,Hl:  can't  compute  for  ne*-  ,type  =  %d.\n‘,neUype); 

exit(-l); 

^  break; 

if(nectype  *=  3){ 

/•*  Compote  outputs  of  second  bidden  layer  *4 
loopKhkfejwo) 

{ 

if(L2jiiask(i]=OFF)  { 

U.Xn]  =  L2.Y(i]  =  0.0; 
continue; 

sum  E  0.0; 

loopj(hidejone)  sum  »  L2.X(j]«L2.w(i](i]; 
sum  *=  L2.theta(i]; 

L3.X(i]  E  L2.Y[i]  ^  sigmoid(  sum ); 


/««  Compute  outputs  of  dual  layer  *4 
loopKnunuoutputs) 

{ 

ifCoutputJlayer— •roask[i)aa>OFF)  { 
out|NitJayer-*Y[i] »  0.0; 
continue; 

sum  >  0.0; 

loopj(hide  Jwo)  sum  outputJayer-«XU]«outpuUayer-»w(j](i]; 

sum  *=  outpuLJa^-»theta(i]; 
outputJayer->Yti]  *  sigmoid(  sum ); 

} 

}  /*  end  compute-outpuLfrom-Hl  4 


void  coniputejoutputJin)mJ12() 


double  sum; 
intij; 


printfCps.c—  compute_output_froiiUI2  \n*); 


ifCneuype  #  3)  { 

^printfCstdm;  'coopute_output_froia_H2:  can't  ccxnpute  for  net.type  =  \ 
td.Xn'.neUype); 
exit(-l); 

} 

/**  Compute  ou^wts  of  final  layer  *4 
loopKnunuMitputs) 

{ 

iKUjnasklil— OFF)  { 

U.Y(i]»0.0; 

continue; 
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sum  >  0.0; 

loopj(liideJwo)  sum  L3  JC[i]«L3.w(j][i]; 
sum  ■»«  L3.theta(i]; 

L3.Y(i]  s  sigmoidC  sum ); 

} 

}  /*  end  computejoutpuLJmnLH2  4 

/*«**«*««*«**«**««****«*««**«***«*«*4 

double  sigmoid  ( a ) 
doubles; 

double  maxjexp; 

maxjexp  >  S0.0; 
if  ( a  >  nuuucxp )  return  1.0; 
if  ( a  <  —maxjexp )  retuni  0.0; 
retum  1/(1  exp(— a)); 

}  /**  end  sigmoid  *4 

/^^^:t‘m**^0*^*m*****t****************4 

double  symmetric^igmoid  ( a ) 

doubles; 

double  maxjexp; 

maxjexp  at  SO.O; 
if  ( a  >  maxjexp )  return  1.0; 
if  ( a  <  —maxjexp )  return  —1.0; 
return  ((2^1  +exp(-a)))-  1); 

}  /*«  end  synunaric^igmoid  *4 

/***^*m****9****»********m**************** 

The  following  structure  definition  and 


’idno-parthiott"  and  ’partition"  fiinctkm 
wbicb  follow 

**«*«««««*«4i*««**««*«««*4>*****«****«****««4 

typedef  struct  srculata  { 
int  vecmum; 

int  wed; 

}  srcjdata; 

MATRlX-ALLOCATOR(srcilata.matiixjtcjdala) 

MATRIXJKEE(srejlata,freejnatrixjn;jlata) 

/*******«*«*«**«****«***««*«4i4i4>****««««««****4>******«**«*******« 

idnet-pattition:  separates  the  dauirase  into  training  and  lest  sets. 

•«**««««***«««4.**4i*«*««**««4.«**«*««**«*««*«********«*********«4 

void  idnet.paititioo(  part-seed ) 
long  paitseed; 

{ 

srcjiata  *«src; 

int  i,j,  class,  cntsO,IDX; 

int  tmjcnt «  0; 

SIC  -  matiix,srcjdata(0,numjoutput8-l,0,num.vectors-l); 

/**  Cdippule  partitkm  *4 
nuiium  «  nuituvectors; 
tmJist  <■  ivectof(0,nunura— 1); 

/**  Set  up  the  clasa  data  in  a  format  amenaUe  to  picking  randanly 
from  each  clast  *4 

loopij(numjoutputs,num.vecton)  src[i]|j].used  ■  False; 
kM^num-vecton)  { 
class  >0; 

sic(class)(i].vecjium  «  vecjium(i]; 


«ifdef  DEBUG 

printft 'Vectors  assigned  to  class  0  are:\n*); 

loopj(nuin.vectofs){ 
pn^*  %d  *,sic[0]U]); 

prinif('\n'): 

#en^ 

/**  Now  actually  assign  the  vectors  to  partitions  *4 

mitsui«(paitjeed,  state,  STATE^IZE ); 

kx>pj(nuin.vectors)  { 
while  (1)  { 

if(  sic[0][(IDX  landomO  %  nuin-vectOfs)].used  =  False )  { 
sic[0]{IDX].used  s  True; 
tniJisUtrnxnt]  =  sic[0][iDX].vecjium; 
tnijcnt-H-; 
cnt++; 


numJm  s  tnucnt; 


«ifdef  DEBUG 

printf('%d  vector  IDs  assigned.  \n*,cnt); 

#en^ 

#ifdef  DEBUG 

fprintf(stderr,  'The  %d  vectors  in  trn_list  are:  '.nunum); 
lMpi(numJni)fprintf(stden,*%d  *,tniJist[i]); 
fpnntf(siderr,*\n*); 

#eiKlif 

fieejiutrixjicjdata(stc,04iumjoutputs— 1.0,aum.vectors— 1); 

}  /*•  end  klaeLpartition  *4 

N*m***************0***^******m****************************m**<v* 
partition:  separates  the  database  into  tndaing  and  test  sets. 

******a******a******************t******************0**********4 

void  partition(  uitpetcent.  pait.seed ) 
double  cntpeicent; 
long  panJeed; 

{ 


sicjdata  *«sic; 

im  nuin.vecs(MAXX>UTPUTS]; 
int  nununiA[MAXX)UTPUTS]; 
int  nunungn(MAXJ[)UTPUTS]; 
int  i,j,  class,  cnt  =  0,IDX; 
int  tni^t  K  0,  tstxnt  >  0; 


SIC  s  matnuicjdata(0,nunijoutputs-l,0,num.vectois— I); 


/•*  determine  number  of  vectors  of  each  class  *4 
loopKnunuNitputs)  nuin.vecs(n  «  0; 

loopKnuiiL.vectors)  num.vecs[nn(Linax(dbjout(i]jiuiiu)utputs)]'M’; 


iifdef  DEBUG 

printfi('The  database  contains  %d  total  vectors. \n*,nun].vectors); 
loopKnumjoirtpats)printf(*  %d  in  class  %d\n*,nuin.vecs[il.i); 


1*0  Compute  partitkms  04 

ktopKaunijoutputt)  nunutmAIi]  >  (intX  tm4ieicent*(double)num.vecs(i] ); 
niiiiiJni>0; 

iifdef  TRN 
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loopKb'jiiijoulputs)  miiiuni  -f*  nunurnA(i]; 
trn^  >  ivect^O.nuiiUni-1); 
iendif 

nunust «  num-vectofs  —  nunurn; 
tsUist  >  ivectoi<04iuiiust- 1); 

#ifdef  DEBUG 

prinl^'The  number  of  vectors  assigned  to  training  by  class  are:\n*); 
k>cpi(nuia.outpuU)printf(*%d  training  vectors  in  class  %d.\n', 

;  numACi],  i); 

#e  if 

I**  Set  up  the  class  data  in  a  fonnat  amenable  to  picking  randomly 
from  each  class  ***4 

loopiKnuinjoutputs,nuin.vectors)  sic(i](i].used  »  False; 
loopKnunijoutputs)  nuinjisgn[i]  =  0; 
loopKnum.vectora)  { 
class  B  findjiiax(dbjoutIi],nunu)utputs); 
sic[class)(aunLasga[classl-M'].vecjium  =  vecjiuin[i]; 

} 


«ifdef  DEBUG 
loopKnunuwtputs)  { 

ftnntfC Vectors  assigned  to  class  %d:',i); 
oopj(num.vecs[i]) 
pnntfC  %d  '.src[i][j]); 
priiitf(*\n‘); 

#axiif 


«ifdefTRN 

/«*  Now  actually  ass^  the  vectors  to  partitions  *4 
initstate(pait.seed.  st«e,  STATE^I2E ); 
loopKiniiiuoutputs)  { 
k>opj(aiifiuniA[i])  { 
while  (1)  { 

if(  sicIi]I(IDX  B  landoinO  %  nuiiL.vecs[i])].iised  =  False 
sn;[ij[IDX].used  » Ihie; 
tniJist(tn*jcntl  *  src[i][iDX].vccjiuin; 
tnxnt-H-; 


){ 


nunUrn  b  trnjcnt; 
#eiidif 


ioopKnum-cutputs) 

k)opj(num.vecs[ij) 

IK  sic(i][j].used  bb  False )  { 
tsUistltstjcnt]  B  src[i]U].vecjiuni; 
tstcnt-H-; 
cnl++; 


numJst  B  tstxnt; 

priiitfl(*\n%d  test  vectors  assigned. '.nuiaist); 


«ifdef  DEBUG 

fjprintfltlden,  ‘The  %d  vectors  in  tm.list  are: 


loopKnainJni)fpfiiilf(tiden;*%d  ‘.trnJistli]); 
ft>niitf(stden','\n*); 

fprinlKsttleiT,  ‘The  %d  vectors  in  tst.list  are: 


lo(^niiiiust)fiprji)tf(stden;‘%d  ‘,  ttUiufi]); 
fpniilf(stden;‘\n‘); 


‘.nunurn); 
‘,  nunUst); 
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iendif 

free  jnatrix  Jic  j(iata(src,0,niiiiijoutputs—  1  .O^um-vectors  —  1 ): 


} 

void  mp3  Jc^er() 

I****************************************************** 

mpSJuifer.  tnmfeis  the  weights  from  the  layered  sductuie 
to  a  vector  stnicture  for  use  with  kalmaa  traiaiog  or  saving  data 
to  a  fiJe. 

«*****«««*«****4t**«*4i*;»«******4r*«*«««»««*«««**««««**«*^ 


im  IDX>:0,  i,j; 

-  The  kalnm  algorithm  assumes  the  weight  matrix  is  of  the  form 

-  W(desustc); 

-  The  thresholds  are  put  into  the  weight  matrix  as  another  column. 

-  Thus  tbe  input  vectois  for  eacb  layer  are  augmented  at  tlie  end 

-  with  an  entry  of  unity. 

-Thekalman  vector  is  stored  in  row  mt^or  form  starting  with  the 

-  first  layer  followed  by  the  second  layer  and  then  the  third  layer. 
•4 


printf(*ps.c--  iap3_k_xfer  \n*); 


loopKhidejone)  { 

^p|(numJnputt)  r^pOX-H-]  >  LI  .w[i][i]; 
kwpKhide  Jwo)  { 

loopKIudejone)  xhatPDX-H-] »  U.wQKi]; 
^xhatllOX-M-]  =  L2.theta(iJ; 

loopKnunuoutputs)  { 

loopj(hideJwo)  xhatPDX-H-I  s  L3.wn](i]; 
xbatllOX-M-]  m  L3.tbeta(il; 


} 


void  mp2JLJifei() 

f**mmm****m*******m»*************************vtti******** 


mp2Jutfer  transfers  the  weights  from  tbe  layered  structure 
to  a  vector  structure  for  use  with  kalman  training  m  saving  data 
to  a  file. 


«*****»*««****«***««****«««**«*««««********«**4i4i«**«**tl 


int  IDXsO,i.j; 

/** 

-  Tbe  kalman  algorithm  assumes  tbe  weight  matrix  Is  of  tbe  form 

-  Widest^): 

-  The  Ibtesboids  are  pot  into  tbe  wei^t  matrix  as  another  cohmut. 

-  Tim  tbe  input  vectors  for  each  layer  are  augmented  at  tbe  atd 

-  with  an  entty  of  unity. 

-Tbekabnan  vector  is  stored  fit  row  mrtfdr  form  startiim  with  tbe 

-  hist  layer  followed  by  the  second  layer  and  then  the  third  layer. 
*4 


printK’ps.c--  i«p2_)c_;c£er  \n"); 
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k>q>i(hidejoae)  { 

lo^|(mm-mpub)  s  Ll.w[j][i]; 


] 


} 

} 


loopKnum^outputs)  { 
loopKhidejoiie)  xiut(IDX-M-]  = 
xhatllDX-H-]  m  L2.th^i]; 


L2.w(j]Ii]: 


void  gaussianjionnalizeO 

^«**«*««**«***«**««***«***«*«***«***««««**« •«•««***•*****«*«**« 

gaussiattJttonnalaeO:  Nrnmalize  data  so  that  the  training  set  has 
a  mean  vector  of  lem  and  a  standard  deviation  vector  of  all  ones.  Only 
tbe  features  bang  used  am  normalized. 

INPUTS:  tumum 
tmJistl] 
vecjntryn 
nuauvectors 
num-inputs 
dbJnnn 

dominant-sensor 

OUTPUTS:  dbJnfJfJ 
meanf] 
sd[] 

**«*«*4>**«««***«***«**««*««*«**4r«««***«4i*«««««4i«#4i]»***««*«****^ 

double  4>suiii,  *sunL2; 
int  vcc; 

int  i,j,  ktlnununputs.  taiget; 

UprintfCpst:-  gaussianjiotmalize  \n’')p4 

sum  s  dvectoi(0,aum  Jnputs- 1 ); 
suin^  «  dvectoKO^unuoputs— 1 ); 

if((doniinantjensora:FliR)||(do(ninaiiUsensors3:RNG))  InumJnputs  numJnputs— 1; 
elK  InumJnpuls  «  numJnputs; 

loopi(InumJnputs)  sum(i]  ^  summit]  >  0.0; 

h*  Compute  mean  and  sd  of  training  data  *V 
k>opi(nunUra)  { 
ta^  ■  tmJistli]; 
loopj(num.vectors) 
if  (taiget  ss  vecjium(j]) 
biw; 
vec*j; 

loopj(lnumJnputs)  { 
sumH]  **•  dbJn[vec](j]; 
sum^Ul  -fB  db  Jn[vec)U]*db  Jn[v6cl(j]; 

} 

} 

loopKInumJnputs)  { 
meanp]  >  sum(i]  /(double)  nunutra; 
sd[il «  sqit(  (sunL2(i]  /(double)  nunutra)  -  mean(i]«niean[i] ); 
if(sd[i)->0.00)sd[il-  1.00; 

} 

«ifdef  DEBUG 
printfT'Means :  *); 

loopKInumJnputs)  piintf(*%g  *,mean[i]); 

plinCK'Nn*); 

printf(*Std  dev:  *); 

]oopKlnuniJnputt)printf(*%9  *.sdn]); 

piintf(*\n*); 

«endif 
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h*  Sow  apply  to  all  data 
loopKnuin.vectois)  { 
toopKInuinJnputs)  { 
dbJii[i][j]  >  (dbJnlilUl-nieaii[j]Vad[j]; 

} 

^ulvectoi(suiii.04>umJapiits- 1 ); 
free  jdvectoi(sumJ,0,nununputs—  1 ); 

}  /**  end  gaussiaa.jiomialize  *¥ 
gaussiattMoammaltBe():  Unaonnalize  data. 


ISPUTS:  aunL-vecuus 
aunuaputs 
dbJnnf] 
dominanLseasor 
meanU 
sdU 

OUTPUTS:  dbJnfln 


•«**«4i*««««*«4i**«**«**««***««*««**««**«***W 

void  gaussunjuanomializeO 

{ 

int  i,j.  Immunptas; 
hpriBtiCps.c-gaussiaajiaaonnaliit  \B");d 

if((doiiiinantjMiisoiteFLIR)||(doiiiiiiaiit^iisof^RNG))  InumJnputs  s  numJnputs- 1 ; 
ebe  InumJnputt  *>  numJnputs; 

/•*  Sow  unaonnalize  aU  data 

loopij(nuin.vectotsJnuiiiJnpuis)  dbJn[i](j]  >  db  Jn[i][i]«sd(j]  *  meanQ]; 

}  /**  end  gaussimumnonnalize  *4 

(mesjionnalize:  Sonaalizes  data  between  I  and  -I. 

Must  be  called  alter  the  above 
faussianJiomialize  routine  is 
invoked. 


ISPUTS:  dbJttOn 
numJnputs 
oum.vectofs 

OUTPUT:  max.value 

********«*4,««««***««««*«*«**«««***««*«««******«*4 

double  onesjKMinalizeO 

{ 

int  U; 

double  max-valuel  ■  0.0; 
double  max.value2  ■  0.0; 
double  max; 

loopKnum-vecton) 

^loopj(numJnputt) 

^if  ((max  B  fd)t(dbJn[i](j]))  >  max.valuel) 
max-vahiel  *  max; 


if  ((max  >  max.value2)  AA  (max  max.valuel)) 
roax.vahie2  ■  max; 
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fjprinlf(RUNJ’n£,‘\nones_nonaalize  inax_valuel  is  %9.61fMnax.valuel); 
fi>riiitfWUNJ’lLE,*\nones_nonnalize  niax_value2  is  %9.61f*.iiiax.value2); 


/«  now  divide  ail  data  by  the  max.  value  d 
if  (mar. value!  #  0.0) 


) 


.  .^i(num-vectors) 
loopj(numJnputs) 

dbJn[i]{j]  s  (dbJn(i](j]/iiBx.vahiel); 


letura  max-vahiel; 

}  /*  end  oaesjonaalize  4 


A.6  psx.c 

/«**«**«******««<»4i4i««**«*«**««4<«««*««***«*******«*«******«*«*******« 

psx.c;  Perceptmn  support  padcage  with  routines  that  are  dependent 
on  the  nuamer  of  layers  in  the  netwoik. 

Dennis  W.  Ruck.  AFTT/ESG 
DS-90D 

^i0**m***m*****************m*m************************************4 


#iiiclude  <stdio.h> 
iioclude  <iiiath.h> 


#iiiclude  <macrosJi> 
#inchide  <globals.h> 


double  sigmokK): 
double  symmetric  jigmoidO; 
void  computejoutput(); 

void  mallocJay^^^^’ 
double  «*dmaTO(); 
double  •dvectorC); 
void  display  JnputO; 
void  display joutpuiO; 
void  displayO; 


void  iniuietO 

JL _ _ _ _ _ 

iniUidO:  Initializes  data  stnictures  depending  on  the  number 
of  layers. 

Input:  neuype,  nuaUnputs,  bideMie.  bideJwo,  ounuw^puts, 
etaJn,  etZMit,  etaJIlJI2 

Output:  LayeifJ,  outpuUayer,  nuaistates 
im  i.j; 

wtsJype  -  WTS.TYPE.I; 
loopi(nuinJnpiMs)  inpuunask(il «  ON; 


hprintK’psxx-  iniuiet  Xu'l.'d 


switch  (neuype)  { 

1:  hidejone  B  hide  Jwo  <■  0; 


ijOiie> 

nunustates  >  niiiiiJnputs*oumjoutputs  *  nunuxitputs; 

outputJayeTB&LI; 

Layei  0]b&L1; 

LayenO]-»nunu)utputs  >  nunuoutputs; 
Layer(Oj-»nuinJnputs  *  numJnputs; 

La^O)-»eu  «  eta^; 

iiiallocJayer(  Layer[0].Layer(0]-»ouinJnpuU.Layer(0]- 
bieak; 

case  2:  hideJwoBO; 

nunutates  >  nuiaJnpuis^hidejOiie  hidejooe 
hidejone^nunuNitputs  nunuMitputs; 
outputJayer  B  &L2; 

LayerlO  b&L2; 

Layeifl]  B  &LI; 

LayerlO  -»nunu)utputs  >  aiinuoutputs; 

LayerfO  -»nuinJnp^  b  hidejone; 

LayerlO  -*eta  b  etauMit; 

Layerll]-*oumjoutputs  =  hidejone; 

Layerll  j-»nuiiiJn|Mits  b  numJnputs; 

Layerll]-*eu  b  etaJn; 

nuulocJayetl  LayerlO]XayerlO]-»nuiiiJnputs,LayerlO]- 
mallocJayetl  LayerlljXayerll]-*nuinJnputs,Layetll]- 
break; 
case  3: 

nunotates  b  numJnputs*hidejDne  *  hidcjOoe  * 
hide  jone*hide  Jwo  *  hide  jwo  -t- 
hideJwoonunuMtputs  nunuMitputs; 
outpittJayer  b  &L3; 

Uy^BAL3; 

Layerll  bAL2; 

Uyerl2  b&LI; 

LayerlO  -»nunLoutputs  b  nununitputs; 

Layer  0  -*nuiiiJnp^  b  hkteJwo; 

LayerlO  -^ea  b  etajout: 

Layerll  -tnuiiLjputpuis  b  hideJwo; 

Layer  l]>-*numJnputs  b  hidejone; 

Layerll  -•etaBetaJHlJI2; 

Layerl2  -•num.outputs  b  hide.ooe; 

Layer  2  — •numJnp^  b  numJnputs; 

Layerl2  -•eta  b  etaJn; 

mallocJayerl  Layetl0],Layetl0|-*numJnputsXayer{0|- 
mallocJayet(  Layetll|Xayertl]->numJnputsXayetIlj- 
mallocJayeK  Layefl2]XayerI2]-»numJnputsXayerl2j- 
bteak; 

default:  fpiintf(stdetT,*init_net;  invalid  net.type  ; 
exW-l); 
break; 


muisjoutputs ); 


•nuououtputs ); 
•nunuxitputs); 


•mmuxitpms ); 
•nuniJOUtixMs ); 
>numjOut|wtt  ); 

%d\n*,  neuype); 


} 

xhat  B  dvectoi(0,nunLatates- 1); 

loopKneuype)  loopK^ayerlil-*  nunuxitputs)  Layctli]->masktj]  b  ON; 
}  /«  end  ittiunet  ^ 


h 


Junpxjtfer  masfer  weights  from  vector 
to  a  layered  structure 


void  lunpxjtfeiO 


iii!l!)X-0; 


/*print/("psxx-  kjnpxjtfer  \n  ”),-¥ 


iloopk(neUype)  { 

ha*  Oet  Next  Layer  **4 
loopi(Liiyerlk]-»namjOutputs) 

{ 
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Ioo|>KL«yerrk]-*iniinJiipul8)  Layei(k]-»w[j][i] «  xhat(IDX-M-]: 
Layo^Hth^t] «  xIu([IDX-h-]; 

} 

} 

}  /•  eadJui4>xjt/er¥ 

l*******************m****m**************t******m**m**** 

mpxJuifer  tnuafas  the  weights  from  the  btyered  stiucture 
toavectorstmcture  for  use  wahkalman  training  or  saving  data 
to  a  hie. 


void  mpxJuifeiO 

{ 


int  IDXBQ,i.j.k: 

-  The  kabnan  algorithm  assumes  the  weight  matrix  is  of  the  form 

-  Widest^): 

-  The  thresholds  are  put  into  the  weight  matrix  as  another  column. 

-  Thus  the  input  vectors  for  each  layer  are  augmented  at  the  end 

-  with  an  entry  of  unity. 

-  The  kalman  vector  is  stored  in  row  major  form  starting  with  the 

-  Srst  layer  followed  by  the  second  layer  and  then  the  thud  layer. 

*4 


/*ptintf(’psx.c-  mpxJutfer  \d  ’);4 


rioopk(iieUype)  { 
loopi(LayefW-»niiinjoutputs)  { 

k)opj(Lay^]-*iiunLJaputs)  xiut(IDX'H-] »  Layef(k]-»wn](i]; 
xluttllDX-H-] «  Uyer(k]-*dieta(i]; 


}  f*  end  mpxJutfer  4 
void  coropmejoutputO 


switdKnetiype) 


1. 

case  2: 

case  3:  conmotejoutputxO; 
break; 

defauit:  printf(*coDipute_ouCput:  can't  perform  calculation  for  net_type 
%d\n*.  neLlype); 

} 

)  head  cort^ntejoatput  4 

/m»*m*m**»»***»******»****mm******m*******»*4 

void  computejoutputtO 


double  sum,  *inask; 
intij.k; 


rloopk(neUype) 

loopKLayer(kl-»iiunLjOutputt) 

if(Layef{k]-*mask{i]nOFF) 
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iflkiCO)  UyerIk-l)-*X[i]  =  0.0; 
continue; 

u(k»neuype— 1)  mask  >  inpuunask; 
elM  mask  >  Layeilk-i-l]-4mask; 
sum  ■  0.0; 

kx>pj(Lawr{k]-»numJnputs) 

sum  •>«  Layer(k]-»X(j]*LayeTPc]-»w[j](i]*mask[j]; 

sum  **  Layer(k]— •th^i]; 

iifdefUNEAROUT 
if{ks=0)  LayerIk]-*Yti] »  sum; 
else  Layer(k)-»  Y[i]  c  sigmoid(  sum ); 
iendif 

«ifdefSYM^IGMOID 
Lawitk]— •Y[i]  k  symmetric  jigmoid(  sum ); 

#endif 

«ifdef  SIGMOID 
Layerlk]— •Y[i]  *  sigmoid(  sum ); 

#endif 

#if(lefINP.SYM 

if(kn(neuype- 1))  Layer[k]-»Y[i]  =  symmetric  jigmoid(  sum ); 
iendif 

if(k#0)  Laycifk-ll-.X[il «  Layer(kJ-.Y[il; 


}  /*  end  con^MCjOUlpulx  4 


h********m******i,*************m****i,********4 
void  malloc  Jayer(  L,  inputs,  outputs ) 
struct  layer  «L; 
int  inputs,  outputs; 


h  pnnti("psx.c-auUocJayer\a“);4 


L-*  w  a  dmauix(0,inputs— 1 ,0,outputs— 1 ); 
L-«dw  a  dmatiix(0.inputs- 1 ,0,outputs- 1 ); 
L->  w  jold  a  dm8lrix(0.inputs—  1 .0.autputs—  1 ); 
L-*theu  a  dvectof(0,output8—  1 ); 

L-»dtheta  a  dvectoi(0.outputs— 1 ); 
L-»tlieta^ld  a  dvectoi(0,outputs- 1); 

L-*beu  a  dvectoi<0,outputs- 1 ); 

L-«gamma  a  dvectof(0,outputs-l); 

L-»del  a  dvectot(0,out^-l); 

L-*madc  a  dvectot(0,outputs- 1 ); 

L-»X  a  dvector(04nputs- 1 ); 

L-»  Y  a  dvectoi(0,outputs—  I ); 


} 


void  display  Jnput(number,  max,  size) 
int  number,  size; 
double  max; 

FIL£«image; 
double  temp; 
inti; 

stauc  int  numJn  a  0; 
cliarinfile(20]; 
char  coniituuid(80]; 


numJn-t-f; 

sprintf(infile,  ‘tstdts*.  *IN*,nuinJn,  *  .rec*); 
image  «  fop^infile,  'w‘); 


loopjCiluinJnputs) 

leiiq)  m  ((<n>Jii(nuiiiberl(j]«iiiax>»sd(j])  meanQ]; 
fpriiitf(iiiiage,  *tl0.ef  ‘.temp); 

/ck)se(iiiu^); 

«ifdef\lEW 

spfintftconiiiiaiid,  *%s%s%s', 'cp  ‘.infile,  *  tempIN'); 
systenKconunaiKl); 

^(fe|)toy(size,  'tav>lN*); 

} 


h*******************************************4 

void  display joutpuKmax,  size) 
double  max; 
intsize; 

FILE  vimage; 
double  temp; 
inti; 

static  int  nunuout  >  0; 
cliaroutfile(20]; 
char  comm^iSO]; 
double  sqitO; 


nunuxit-t-^; 

sprintfi(outfile,  *%s%dts*,  *OUT*.numjout.  *  .rec*); 

image  >  fopen(outfile,  'w'); 
loopKnumjout|>uts) 

^temp  >  ((Layer(0]-»  Y(i)«max)*sd(i])  *  mean[i]; 
fptintKimage.  *%10.6f  ’.temp); 

/ck)se(iniaae); 

#ifdefVlEW 

sprintf(command,  *ts%s%s*, 'cp  '.outfile,*  teiq)OUT*); 
systemCcommand); 

^(faj;lay<size.  *tei^UT*); 

} 


h*****************************m*****m 

*  displMyx 

*  coavats  a  .am  6k  to  rk  tad 

*  dispkys  it  m  openwiadows  using 

*  xli. 

void  display(  X,  filename) 
intX; 

char  filename^; 


char  command[80]; 

spfintKcommand, '%s%s%8*. ‘cp  ‘.filename,*  tenp.rec*); 
systemCcommand); 

system(*float_gray  temp. rec  temp. red'); 
sprintfCcommand,  '%8%d  Id*,  'graytorle  -o  temp.rle  *,X,X); 
strcatCcommand, '  temp. red'); 
systemCcommand); 

sptintf{oommand,*%8%s%8'.*mv  teBV).rle  ', filename.*. rle*); 
systemCcommand); 

sprintKcommand,‘%8t8%8*.*xli  -quiet  -zoom  300  '.filennoe,  *.rle  &*); 
systemCcommand); 

} 
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A.  7  utils.c 


A************************************************* 

Utility  Fuactioas 

**m**********»**************»*******************4 

iinclude  <sldioJ» 

iifndefNEXT 
#iiiclude  <iiiallocJi> 

#else 

exteni  char  *  mallocO; 
iendif 

#iiiclude  <n]alhJi> 

#include  <stiing±> 

#iiiclude  <iiiacn».h> 

#iiich]de  <globals.h> 

/**  External  System  Calls  *4 
extern  char  *  getenv(); 
dwble  dranw  (): 

/•*  Jhtemai  Functions  *4 
int  fiixLmaxO; 

Boolean  find(); 
void  systemjcheckO: 
void  soitO; 
char  *  furse  JiuuneO; 
char  •  geuoken  (); 
double  gaussianO; 
double  distance20; 
char  *  makejuuneO; 

/*ta**a*0**a**«*a*a*a****a**a****a****»*****4 

int  find  Jiiax(  data,  len ) 
double  •data; 
int  len; 


double  max.val  «data; 
int  max  Jdx  •  0; 
inti; 

loopi(len)  if  ( •(data+i)  >  niax.val ) 

niax.val »  •(data+i); 
maxJdxsi; 

letum  maxJdx; 

}  /•  end  Bndjnax  4 

h********************************»********»4 

voidsntenucbeck() 

/••  Checks  the  sydan  /or  leiiable  operations  *4 

^static  char  scjiaine[801 «  * sy8ten_checklA992* ; 
FILE  •«; 

double  xjout- 123321; 
double  xJn; 


printf(*util8.c--  8y8tem_check  \n*); 


/*WCbeck4 

iK  (tc  «  fopen(tcJiaine,  *w*))  NULL) 

{ 

fjptintK stderr,  *sy8teQ_check:  can't  open  ts  for  writing.\n\ 
FATAL  Error  An*,  tcjuune ); 
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exil(-l); 

} 


feriiitf(  sc,  REALJ’MT,  xjout ); 
fclose  (sc); 


ifC  (sc «  fopenCscJuune,  *r*)) »  NULL) 

fjpniitf(stdeiT, ‘systen_check:  can't  open  %s  for  reading. \n\ 
FATAL  Error .  \n*,  scjuune ); 
exit(-l); 

} 

ftcanfC  sc,  REALJ’MT,  &x  Jn ); 
if(  xjout  xJn ) 

{ 

fprintf(  stdeiT,  *systeiiL.check:  Floating  Point  I/O  Error. \n\ 
FATAL  Error. \n’ ); 
unlinkC  scjume ); 
cxit(-l); 

} 

unlink  ( scjiame ); 

printK  ’systenucheck:  OK.Xn" ); 

}  /*  ead  systeaucbeck  4 

I**  Procedure  to  son  an  input  array  *4 

void  (^partition  ( data,  rank,  split,  lower,  upper,  new  J,  newju ) 

double  «data; 

int  vrank; 

register  int  split: 

int  lower, 

int  upper, 

int  vnewj; 

int  •newju; 


register  int  i  s  lower,  j  x  upper, 
register  int  temp; 


printR'utils.C”  Qpartition  Xn"); 


do  { 

while  ( datalrankli]]  <  data[split] )  iH-; 
while  ( datalsplit]  <  dataln^Ij]] )  j — ; 

temp  a  tank[i]; 
rankm-nnkUl: 
ranklj] « temp; 
i++;j — ; 

}  while  (i<j); 

*newJ*j; 

•newji  s  i; 


/«*  Recursive  qukkson  *4 

void  quicksortC  data,  rank,  upper,  lower ) 

double  adata; 

int  arank; 

int  upper,  lower, 

^int  split; 
im  new  jip.  newJow; 

int  temp; 
im  i; 


ptintf(*util8.c—  quicksort  \n'); 
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if  ( upper— lower  >  1 )  { 

Milt  s  rank((upper-flowery3; 

Qpattition(  daUk  rank,  split,  lower,  upper,  &new  Jow,  &newjup ): 
quicksoit(  data,  rank,  upper,  newuip ); 
quicksoit(  data,  rank,  new  Jow,  lower ); 

}  else 

if  (upper  —  lower»  1) 
if  ( data(rank[upper]]  <  data(rank[lower]] )  { 
temp  « iank[upper]; 
rank(upper]  >  rank[lower]; 
rank(lower] » temp; 

} 


void  soit(  data,  nunuelements,  rank ) 
double  *data; 
int  numjelements; 

int  *fank; 

L*  Tbe  output  is  the  rank  of  integers,  rank,  wbJcb  gives  the  sorted 
Older  of  die  the  rank  of  doubles,  data.  That  is,  the  smallest  entry  in 
the  array  data  is  index  rankfO].  i 

int  i; 


printfC utils. C--  sort  \n*); 

loopi(oumjelements)  tank(i] » i; 
quicksott(  data,  tank,  nunuelements—  1, 0 ); 


lm***m**********m*******mm**mm******tf******m******m»*m****4 

char  *  oarseJnameC  fname ) 
char*iiiame; 

char  «  ouuiame; 
char  varjiame[256]; 
char  ouUemp{2S6]; 
char  *  var,path; 

int  fnJen,  vn Jen,  vpJen,  ouUen; 
int  i; 


ptintf(*utils.c—  parse.fnane  \n'); 


fnJen  «  strlen(fname); 

h*  See  if  it  starts  with  a  dollar  sign  *4 
if(  fname(0]  »>'$') 

I**  expand  environment  variable  *4 
i»  1; 

whil^  fname[i]  ^  '  / ' ) 
varjiame[i— 1] «  fnamefi]; 


varjiame[i-l]B  '\0'; 

if(  (var-path  ■  getenv(var JMme)>  ^  NULL) 

ptintf(*parse_fname:  environment  >.iable  $%s.  Not  defined*. 

varjiame); 

exit(-l); 

}**  Now  combine  into  the  fiiU  path  *4 
vn  Jen  *  strlen(varjiame); 

VpJen  B  strien(var.q>8th); 
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A'opKvpJen)  ouuefnp[i]  >  var.path[i]; 

for(iaO;i<fn  Jen— vn  Jen—  1  J-h-) 
ouUeinp[i-i-vpJen]  fnaine[i-»’vLkn-i-l]; 
ouuetnpii-fvpJen]  =  '  \0 ' ; 


h*  com  Japat  path  to  output  path  *4 
loopK  fnJen-t-l )  ouUemp[i]  s  fnanie[i]; 


/**  Now  allocate  nemoxy  (or  ouLname  *4 

if(  (ouuuune  =  inaUoc(  (ouUen  s  str]en(outJemp)Hl ))  =  NULL) 

{ 

printf(*parse_fname:  out  of  memory. \n*); 
exit(-l); 

ioopKouulen-f  1)  outJiaiiie[i]  =  ouuemp[i]; 
return  out  jiaine; 

}  /*  end  parse  Jname  4 

char  *  geuokenf  str ) 

FILE*str, 

/****«**4i********«**«*««*«********«<*****«*4i********«««***«*^ 

/• 

Returns  the  next  string  of  characters  in  stream,  STR,  which  is  separated 
with  white  space. 

*****>«>*«««««***«*««**«****«*««**««*******««*«***««*«*«*******^ 


char  temp[1024]; 
char  *  tk.^, 

int  i; 

/**  Find  first  character  of  token  *4 
while(l) 

^ifi(  (teinp[0] «  fgetc(sU’)) «  EOF)  return  NULL; 
if<  (tempiO]  /  '  ' )  &&  (tenip[0]  ^  *  \n ' ) )  break; 


whiled) 

temp(i]  =  fgetc(str); 
switch  (temp[i]) 

{ 

case  ' 


case  '\n': 

break; 
case  EOF: 

break; 
default  : 

} 

if(  teinp[i] 


ttmplil*  '\0'; 
tempHl-  '\0'; 
break; 

>  '\0' )  break; 


}  ’ 

iff  (tluptr  «  mallocC  sti1en(temp>f  1 ))  NULL){ 
fprintffstderr, 'get.token:  out  of  memory. \n*); 
exit(-l);} 

loopK  strien(temp>f  1 )  tk.ptT(i]  -  tempfi]; 
return  tk.ptr. 


} /*  end  geuoken  4 
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double  gaussianC  mean,  var ) 
double  mean,  var, 

Itm*************************************************************** 

gaussian:  Ktums  a  gaussian  randon  variable  sampk  with  specified 
mean  and  variance.  The  centra/  limit  tbeomm  is  invoked  to 
generate  the  sample. 

*m***************************************************************4 


int  numjvs  =  20,  i; 

double  sum  >  0.0,  ave,  norm,  Z,  Y; 


-  Obtain  a  sum  of  random  variables  that  are  uniform  between 
-Oandl. 

***¥ 

loopi(nunLrvs)  sum  -fs  drafid48(); 
ave  -  sum  /(doubleXnumjvs); 

/•• 

-  AVE  isarv  with  mean  s  0.5  and  variance  =  I/(I2*nuitLrvs): 

-  now  normalise  AVE 

•¥ 

Z  =  (ave— 0.5yajft(l. 0/(1 2.0*(doubleXnumjvs))); 

/•* 

-  Now  unnormalize  to  desired  mean  and  variance 

*¥ 

Y  =  mean  -v  sqrt(var)«Z; 
tetuni  Y; 


/*«***««««*««**««««*«**viv*««*«**«*iv«««******««*«****v**«¥ 

char  «makejname(  num,  root,  ext ) 
int  num; 
char  vroot,  *ext; 

*«*«*•*•**«*«««* 

makejiame:  Function  to  create  a  file  name  given  the  root, 
a  number,  and  the  extension.  The  file  name  is  of  the  form: 

root  number ext 

«**««•«•«••«**«*•••****«****«******•***«•****«««***««•**¥ 

char  *fhame,  numJmage[80]; 


printf(‘utils.c--  make_name  \n’); 


sprintf(  numJmage,  *%d*,  num ); 

fname  ■>  malkxX  strlen(roo(Hstiira(numJmageHstrlen(ext>i-2 ); 

strcpy(  fname,  root ); 

sticat(  fname,  numJmage ); 

sucat(  fname, ' . ' 

stroaK  fname,  ext ); 

return  fname; 

}  /*  end  makejiame  ¥ 


char  *  make  Jile(  root,  ext ) 
char  *root,*ext; 


maiceJile;  Function  to  create  a  file  name  given  the  root 
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gnd  the  extensiott.  The  fik  name  is  of  the  fom: 
root  "."ext 

«««**«*«*««*****«**«*««******««********«*****»«***«««*«*^ 

char  *fiiaii)e,  numJinage[80]; 

fnaine  =  maUoc(  strleii(root>fstrien(ext>t-2 ); 
sbcpyC  fhame.  root ); 
strcat(  fname, ' .  * 
sticat(  fname,  ext ); 

return  fnaine; 

}  /*  end  makeJile  4 


Z******************************************************^ 

Boolean  find(iun  Jile,  var Jiame) 

FILE  *iunJile; 
char  «varjiaine; 

int  cnt,  varJength; 
char  str[80]; 


printf(*utils.c--  find  \n‘); 


varJength  »  strlenfvar juune); 
while(l)  f 

if((str(OMgetc(nin  Jile))=EOF)  break; 
if(str(0)«varjiaine[0])  { 
cnt=  1; 
whiled)  { 

IK  ((str(cntKge<c(ninJile)>»EOF)  || 
(cnt>varJength) )  break; 

^cnt-i^; 

stifvar  Jength]  >  ‘\0'; 
if(strcmp(str,var Jiaine>»4))  return  Tnie; 


return  False; 

}  I**  end  find  *4 

double  distance2(  xl,  x2,  len ) 
double  xl[],  x2[]; 
int  len; 

^•**«*«***««««*«**««««««**«*«««««*«««««««*«*«*«****«««*«*««**** 

distance2:  compute  Euclidean  distance  between  to  vectors 

double  sum; 
int  i; 


printf<*utils.c--  distance2  \n’); 


sum  >  0.0; 

loopKIen)  sum  +*  (xl[i]-x2[i])*(xl[i]-x2(i]); 
return  sqrtfsum); 

}  /**  Old  distance2  *4 

MfdefNEXT 
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double  drand48() 
long  Ival; 

Ival  >  randoin()%2 147483648; 
return  (double)lval/21 47483648.0; 

} 

void  srand48($eed) 
long  seed; 

randonKseed); 

lendif 


A.8  display.c 

h************************************ 

*  daplay.c 

*  converts  a  .fia  Sle  to  tie  and 

*  displays  it  in  openwindows  using 

*  xli. 

**#*«*«****«*«***«**«*«4i*««««««««««««4 

void  displayCdimension,  filename) 
int  dimension; 
char  filenamed; 


char  command[80]; 

sprintf(command. '%s%s%s*,  *cp  *,  filename,*  .gra  temp.rec*); 
systenKcommand); 

system(*float_gray  temp.rec  temp. red*); 

8wiich(dimension){ 
case  128: 

syslenK*  gray  tor  le  -o  temp.rle  128  128  tenv^red*); 
break; 
case  64: 

system(*graytorle  -o  temp.rle  64  64  temp. red*); 
break; 
case  32: 

sy8tem(*graytorle  -o  tenv^rle  32  32  temp. red*); 
biw; 
default: 

piintf(*I  don't  know  what  size  the  gra  image  is.*); 

system(*rleflip  -v  -o  hold.rle  temp.rle*); 
sprintf(command,*%s%s%s*,*mv  hold.rle  *. filename, *.rle'); 
systemCcommand); 

sprintf(command,*%s%s%s*,*xli  -quiet  -zoom  300  -smooth  -smooth  *,  filename,  *  .rle&*); 

systenKcommand); 
system('rm  *.red*); 
system(*rm  temp.**); 
sy8tem(*rm  *.rle*); 

} 
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A.9  ghbals-h 

GJobai  \4uiaUes 

int  nuiaJn|wtt.  hidejone,  hklejwo,  nunuxitputs,  tempuxitpiits; 

double  etaJn,  etajwt,  etaJllJI2; 

intneUype; 

int  wts  =  WTS.TYPE.1 ; 
int  numJayeis; 
int  iterations  >0; 

int  initiaLseed,  pait-seed,  tm-seed; 

int  nunLstates; 

int  num-vectors  >  0; 

int  «ftr Jist; 

int  *vecjiuin; 

int  «vecjentty; 

int  nunUrn  >  0,  *tm  Jist; 

int  nunust  s  0,  *tstJist; 

intnonaiype; 

int  dominant-sensor  *  0; 

int  nuiailir,  *flirJist: 

bit  nuiiLing,  *rng  Jist; 

ini  uifienuvector, 

int  max  iterations,  outpuUnterval; 

int  batchiize  a  1,  batchiatt  0; 

char  state[STATE^IZE]; 

double  trn  Jirac; 


double  alpha; 

double  max.value; 

struct  layer  { 

int 

nuiiunputs; 

int 

nunuMitputs; 

double 

eta; 

double 

♦•w; 

double 

••dw; 

double 

••wjold; 

double 

•theta; 

double 

•dtheta; 

double 

•thetajold; 

(kwble 

•del; 

double 

•beta; 

double 

•gamma; 

double 

•mask; 

double 

•X; 

double 

•Y; 

}L1,L2,U; 
double  *inpuunask; 
double  *xoat: 

double  Djout(MAXjOUTPUTS]; 
sinict  layer  *outpuUayer,  *Layi^3]; 
double  *«^Jn; 
double  **dbjoi^ 
double  ««idjout; 
double  •mean,  •sd; 

/••  Gtobal  Data  for  Kalman  Training  •¥ 

double  ••R,d[MAXJOUTPUTS],z(MAXJOUTPUTS]; 


A.  10  macros.h 

h**************************************************** 

Convenient  Macros  for  Perceptmn  Package 

h**  MACROS  **4 
«define  REAL  float 
«ifndefGCC 

tdefine  INTJMAX  (2147483647) 
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#else/*  GCC¥ 

#iix;lude  <Iiinits.h> 

^endif 

tifdefVMS 
idefine  unlink  delete 
#endif 

#ifdefLEO 

«defineREALJFMT*%g‘ 

4telsc 

Mefine  REALJ^  'klg* 

#endif 

«ifdefNEXT 

#undefREALJMr 

Mefine  REALJ^ ‘%lf 

ttendf 

«ifdefVMS 

«undefREALJnMT 

«defineREALJ^*%lf 

tiendif 

#define  Boolean  int 
Mefine  False  0 
MefineTrue  1 

/**  Dominanl  Sensor  Defittidoas  *4 
#define  SINGLED 
«define  FUR  1 
#define  RNG  2 


!**  Mask  DeSaidoas  *4 
«defioeOFFO.O 
«defineON  1.0 

char  junLjesponse[256]; 

#defiae  fskipJine(A)  fjgetsOunLiesponse,  236,  A) 

#defioe  skip  Jine  oet^unk^ponse) 

#define  rtoopi(A)  foiwA)— lu^Ou — ) 

Mefine  rloopj(A)  foi^A)— Id^O'j — ) 

#define  rloopK(A)  foivki[A)—  IjE^Odc — ) 

Mefine  rk>opl(A)  foi(l-(A)-ia>TFa — ) 

Mefine  rtoojmiCA)  foi^nMA)— nin>0w^i — ) 

Mefine  rloopn(A)  for(ns(A)— l^iXTui — ) 

. p50;;) — ) 

-)foi(HB)-ia>0a — ) 


#define  loopj(A)  foi(H)'J<A  j-M') 

#define  loopk(A)  fonk^.k<  Adt-M-) 

#defiae  loopl(A)  foi(M)d<Ad-M’) 
idefine  loopin(A)  foir(nii4)*,in<AmH-t-) 

Mefine  loopn(A)  fof(natO'4i<Aui-M-) 

#define  loopp(A)  foi<pi4);p<A;p4-f) 

Mefine  loopiKA3)  foi<i«0'4<A;i-H-)  foiQaO'j<BJ-H-) 

Mefine  M  ALLOC(A3.C.D)  if((A-(C  *)iiia]loc((B)*sizeof(C)))--NlJLL)  {  \ 
fprintRstderr,  sticat(D,' :  insufficient  ineinory\n*));\ 
exit(-l); } 

Mefine CREATE3ILE(A3.C)if((A-fopen(B.'W))  — NULL)  {  \ 
printRsticaKC,' :  can't  open  for  viriting  -  %s.\n*)3);\ 
exit(-l);} 

Mefine  0PENJ1LE(A3.C)  if((A«fopeii(B.'  r ‘ ))  —  NULL)  {  \ 

printf(sticaKC,* :  can't  open  for  reading  -  %s.\n*)3);\ 
exit(-l);} 

Mefine  idx(U3)  (I)*(N)+(J) 

Mefine  FEQ(A3)  ( ( ((A)-(B))  <  lE-6 )  7  1 :  0 ) 

Mefine  lAR^A)  ((WX(-(A)<(A))7((A)):(-(A)))) 

h*  Dividingby  lOOiasimstbatccaadgccgiyesanKtesidtsmd 
Mefine  IRANI(A)  ((intXnnil(A>»(floatW3<AX)/100) 

/**  All  of  these  are  depeadau  on  the  deSnkioa  of  "layer"  *4 
lAXJNPUTS  1500 


Mefine  M/ 

Mefine  MAX J40DES 


SO 
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«deliiieMAXJllJ40DES  50 

tdefine  MAX  JI2J40DES  50 

«defiiieMAXJOirrPirrS  1500 

MdefineMAX-VECTORS  10000 

MefiiieSTATE.SIZE  256 

«defiiie  WTS.TYPE-MSF  2  haew  weights  Sk  4 
#define  W1S.TYPE.I  I  !*  new  wdgbis  Sk  4 
«defineWTS.TYPEjO  0 /*  oU  weigitls  Sk  4 

«defiiie  TRAIN  0 
MefineTEST  1 

«defiiKTHRE£XAYER  3 
UdefineTWOXAYER  2 

Mefine  MATRIX>LLOCATOR(DATA.TYPE.FCN  J4AME)  \ 

DATA.TYPE  ••FCN  J4AME(niljulMicUKh)  \ 
int  \ 

{\ 

DATA-TYPE**m;\ 

\ 

in>(DATA.*nfPE  **)  inaIloc((uiaigi)ed)  (niti— iiri-»-l)*si2eo^ATA.TYPE*));  \ 
if(!iii)nrerroi<*allocation  failure  1  in  matrix! )*);  \ 
m  -*  nrl;  \ 

\ 

for(isnri;i<nili;i-H-)  {  \ 

iii(iHDATA.TYPE  •)  inalloc((uiuigiied)  (nch— iicl+I)*sizeof{DATA.TYPE));  \ 
if(!fii(i])nferrof(*allocation  failure  2  in  matrix!) *);\ 
inlil-«iicl;\ 

}\ 

fetufnm:\ 

} 

Mefine  MAT1UX  J^REEfDATA  JTYPEJFCN  JIAME)  \ 


void  PCN  J4AME(nuirl,nrtt4tcUtch)  \ 

DATA.TYPE  ♦•m; 

\ 

int  nrl4irit4tcl,nch; 

\ 

{ 

\ 

inti; 

\ 

\ 

-)  free((char*)  (in(i]-MtcI)):  \ 

foi(isnrh;i>ntl;i— 

free((clur*)  (nH-ori));  \ 


A.  11  globals.h 

/*•*«  Global  \knabks  **4 

extent  int  nuinJni^,  hidejone,  hide Jwo,  aumjoutputs,  tenip jxitputs; 

extern  double  etalin,  etajout,  eULlllJI2; 

extern  int  neuype; 

extent  int  wts  Jm; 

extent  int  numJaym; 

extent  int  iterations ; 

extern  int  initial  seed,  panjeed,  tm jeed; 

extern  int  nuiajtales; 

exton  int  nunt-vecton ; 

extern  int  *ftrJist: 

extern  fan  ^vecjmin; 

extent  int  avecjentty; 

extent  int  nunum ,  atmJist; 

extern  int  nunust .  atsUist; 

extern  int  nomtJype; 

extent  int  dominnA  .sensor ; 
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extern  int  ouinJlir,  *flirJist; 

extern  int  nuiiLin^  *ingJist; 

extern  int  current.vector, 

extern  int  max-kerations,  outpuUnterval; 

extern  int  batch  jize ,  batchjcnt ; 

extern  char  ttale(STATE^IZE]; 

extern  double  tm  Jnc; 
extern  double  alpha; 
extern  double  max.vahie; 

extern  struct  layer  { 
int  numJnputs; 

int  numjoutputs; 

double  eta; 
double  «*w; 
double  **dw; 
double  ••wjold; 
double  *theta; 
double  *dtheta; 
double  *thetajold; 
double  *del; 
double  *betv 
double  ^gamtna; 
double  *mask; 
double  •X; 
double  *Y; 

JLI.UU; 

extern  double  *inpuLmask; 

extern  double  •xhat; 

extern  double  DjQut[MAXJOUTPllTS]; 

extern  struct  layer  «out|)utJayer,  •Layr^S]; 

extern  double  **db Jn; 

extern  double  **dbjout; 

extern  double  •*idjout; 

extern  double  «mean.  «sd; 

h*  Global  Data  for  Kalman  'naiains  *4 

extern  double  *«R.  d(MAXJOUlfUTS].  zIMAXjOUTPUTS]; 


A.  12  makedatax 


SAME:  lOBkedtULC 

IsyOKED:  mafcBdara  classUst  Usuize  tmagesia  4jclasses  ouiputfiie 
DATE:2SMn92 

DESCRIPTION:  This  routine  geaeates  the  datt  file  used  by  the  network. 

WRITTEN  BY:  Dennis  L  Kiepp 

MODIFIED: 

SUBROUTINES  CALLED: 

FUTURE  MODMCATIONS/BUGS: 


iinchide  <sldioJi> 

fiinclude  <niathJi> 

tNnclude  <sidlibJi> 

iinchide  <slrinn±> 

ideline  PIX^  1024 

/****«•««*•******••*•«**««**•«■ 

1b  use  lamer  imises  change  the  siae  of 

PIXELSSbme. 


fattimageCnXELS]; 


maiii(arBC4>Bv) 

httatgc; 

char«atgv(]; 


FILE  efnaein,  afout,  *clanfile; 
im  dau.M.k.J; 
im  clast.  I,  inpiat,  outputs; 


4 


<4 
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inttenm; 

chtf  autlile(30],  fileiiame[30].  tetnpfile(30]; 
char  *sticpy(); 

If  (argc  #  6)  { 

printlC ! ! !  The  conanand  line  should  be  ! ! !  :\n\n  makedata  clas8list_file  f_iiles_in_li8t  Image.size (pixels) 
i_classes  Output_file_naine\n*): 
exit(0); 

]***mt*********m***  Set  Up  Files  ****«**’************m***4 

if((classfile-fopen(ai8v(l],  'r'))«a>NULL) 

printfCI  can't  open  the  classlist.file*); 

fHusIKstdout); 

exit(-l); 

} 

M>atoi(aigv[2]);  /*M*  Sumber  of  images  in  dassUstJie  4 
inputs  >  atoKatsvP]); 
oittputs  «  atoKaifv[4]); 
strcpy(outlile,  afSv[S]); 


/•*•*•*  Open  output  Sie  for  writing 

if  ((fout  -  fopen(outfile,*w')) «  NULL) 

{ 

printfiCl  can't  open  the  output  file  %s  Nn'iOutfile); 

tHusiKstdout); 

exit(-l); 

} 

piiniff' Output  file;  %s  \n\n''.otitfile);fHush(stdout): 

ftvintfCfout,  *%d\n'4nputs); 
fprintfCfout.  *%d\n*.  outputs); 

/•a********************************************* 

1.  Read  dataJistMe  for Slename 

2.  Write  exemplar  number  to  output  Sle 

3.  Copy  input  file  to  output  fife 


fof(k>l;  k<M;  k-M') 

{ 


fscanf(classfik,  ‘tsVn*.  filename); 

printff* Input  file:  %s  \n*,  fifetiaine)',fnusb(stdout); 

fscanf(classfife,  '%d*,ftclass); 
printf(*Clas8  is:  %d\n*. class); 

fprintfffoin,  '%d\n',k); 
printfCExenplar  nuober:  %d\n\n*,k); 
tfluslKitdout); 


if  ((lacein  >  fopeaffilename,  'r''))» IfULL) 

printf(*I  can't  open  the  input  file*);niush(stdout); 
exit(-l); 

} 

while  (ftcanfffaoein.  ‘hd*.  Atemp)  ■■  1) 

^irintfffout, '%  8.4f  *,(floai)ieii9); 

} 

/****  Write  dan  data  to  fife  ***4 
fof(H)iitputad>ld — ) 
if(claBS— J) 
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lprintll[foul,*%  f  •.0.90000); 
ifCclass  9  j) 

fprintf(fout,*%  f  *.0.10000); 

} 

fi>riiilf(fout.  ’Is".  "VnXn*); 
fciose(faceiii); 

} 

fclose(fout); 

fclose(classfile); 

prilltf<*\n!  !  ! !  !  AU.  DONE  ! !  ! !  !\n\n* 

} 
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